How I enabled faster local development at Civalgo
Published at: 3/31/2024, 12:00:00 AM
The quest to removing friction in local dev environment
A New job
When I started working at Civalgo in May 2023 I realized one thing, local development was not frictionless. We could argue that a frictionless development environment doesn’t exist but this is not our goal today. Having somewhat a stable and faster dev environment was what we were aiming for.
Civalgo is a construction software company that offers a management solution for all project needs. The company was founded in early 2018 by students from l’ETS in Montréal, Canada. The idea is brilliant and offers a lot to teams in the field.
The stack, the history and the culprit
The stack chosen in early 2018 by those students made sense for its time (we shall only focus on what matters here): Next.js, Tailwind using twin-macro, and styled components. This all seems like a pretty normal stack. The culprit itself only is a culprit because there is something better. Babel was used to compile twin-macro classes into css object and do the heavy lifting for the library. There was also another babel plugin used for having styled components rendered on the server using SSR. Well, in 2018 this all made perfect sense babel was a heavily used compiler and was performant, it was the compiler for next generation Javascript, until Next.js announced in 2021 that they would used a new rust based compiler built on top of SWC. SWC became the norm in Next.js and dev could see a definitive improvement over using Babel.
Why it mattered
Civalgo was going strong for 5 years straight. They went through multiple phases of iterations and developments. The team was building a strong foundation. The codebase wasn’t fully optimized and most of it went through different hands and different levels of experience through time. But, one thing that was always on my teammates lips when I started working there is how often they would have to wait for HMR and Fast Refresh. Everyday I would come in to work and people would complain that it was taking too long. Just getting the initial build from our local dev environment could take around 20 seconds and let’s not talk about the 5-10 seconds HMR/Fast Refresh when updating local code.
Yeah yeah, most of the components, the hooks and the data fetching were not fully optimized for the complexity of the app. But one thing that people were always discussing is how removing Babel could probably help because we could utilize the built in Next.js compiler and get faster local HMR and builds. The focus here was to actually start writing code and get instant feedback instead of waiting for things hanging every time you would change one character and save. There was a number of things we could have done but this was the most discussed one.
No mountain too high
When I started looking at the issue I understood why people were talking about it but no one was actually taking to the time to do it. We were optimizing our current SaaS features and making clients happy which in reality matters a bit more than local performance until it becomes a burden. Babel was primarily used because we had twin-macro and styled components. I mean the styled components was a small issue because after digging through our codebase I realized that we were not really utilizing the SSR benefits of styled components and that removing the plugin would probably take a few seconds. Although looking at twin macro and how many components we had was a bit overwhelming. My first few research made me realized how big the mountain was. But was the mountain really that high…
I went through my first research images just to give you a broad idea of where and how twin macro was used:
To give you a rough idea this is what I had to work with.
A few hundred files, a few different ways of using twin macro and styled components (we will go over ways of using those later on). So after seeing these results I started my research on how I would tackle this and was it even possible to conquer that mountain.
A first hope
After a few searches on the streets of the internet I stumbled into this blog post from Alberto. This was a relief, someone was actually able to do it. The first thought that came into my mind was well it’s Railway; They probably have the resources and the money to actually concentrate on this specific issue but whatsoever I had hope. I started digging into the article and it confirmed a few thoughts on why we want to do it.
In the first few lines of the articles you can already understand that it did massively improved their dev environment performance and made even their CI pipelines fasters.
We’re seeing pages that used to compile in dev in multiple seconds now load in <1s and CI builds are down to 2-4 mins from 4-6 mins previously. In addition, in our testing cold starts in the dev environment improved from ~35s to ~1.5s. Since Twin was the last reason we needed to transpile our monorepo using Babel, this migration also unlocked the last piece of our move over to SWC.
I then kept reading and stumbled onto the why’s and it pretty much confirmed where we were at also. The timeline, the new libraries, the old libraries, the new compilers, the solutions found in previous point in times. All of this was properly aligning with our methodologies and our values. How things degrades over time and that is just common knowledge in the programming world. They even reiterated on the performance which was mind blowing to see and made us more eager to actually make the change.
Now I didn’t have to prove it to the team, part of the work was already done and some bigger companies proved it well. But how could we even pull this out with the number of files we have and with the little team we have. Note that we were only 9 developers working at that time and only 5 full time devs. The other 4 being part time interns that we had to coach and help to keep their growth steady.
Climbing the mountain
To properly understand the next part of this chapter we need to understand how we used styled components and twin macro inside our components. There is three specific way that were most commonly used across our codebase:
The first way of writing our twin macro was pretty straightforward, we used some html element with tailwind classes passed in as string.
// FirstWay.jsx
// ...other code
<HeaderLabel>
<ProjectCode>#{props.project.code}</ProjectCode>
<ProjectName>{props.project.name}</ProjectName>
</HeaderLabel>
// ...
const HeaderLabel = tw.div`border-b my-2 py-2`;
const ProjectCode = tw.div`text-sm text-gray-600`;
const ProjectName = tw.div`text-base`;The second way was a bit more complex, we would use interpolation of twin macro and styled component and this made the string a tad more complex to understand and, we will get there, parse.
//secondWay.jsx
//...other code
<Container {...innerProps} fontSize={fontSize}>
{children}
</Container>
//...
const Container = styled.div<{
fontSize: string;
}>`
${tw`text-gray-800 truncate p-1 pl-2`}
font-size: ${(props) => props.theme.fontSize[props.fontSize]};
`;The third and last way added even more complexity because sometimes we would spread props per css properties, sometimes we would interpolate it inside the string and sometimes we would have nested selectors so it became really hard to make the distinction between the multiple ways we were actually writing those element. There was even more edge “casy” way of doing those but this is enough examples for you to get the gist of it.
//thirdWay.jsx
export const MilestoneTag: React.FC<Props> = (props) => {
return (
<TagWrapper title={`${props.label}`}>
<Key small={props.small} color={props.labelColor}>
{props.label}
</Key>
</TagWrapper>
);
};
const TagWrapper = tw.div`flex items-center leading-6 max-w-xs truncate my-1`;
export const Key = styled.div<{
color?: string;
small?: boolean;
}>((props) => [
tw`border px-2 truncate max-w-xs text-sm`,
tw`rounded-lg`,
props.small ? tw`text-sm` : tw`text-base`,
isBackgroundDark(props.color) ? tw`text-white` : tw`text-gray-850`,
props.color
? css`
background-color: ${props.color};
border-color: ${props.color};
`
: tw`bg-blue-500 border-blue-500
]Now the hard part, the hard part was to try replacing those css properties, twin macro objects or styled component way of doing things into the simplest form we could want: “native” html element with tailwind utility as className strings. That simple.
So to give you an idea, the first example would results in :
// FirstWayTailwindOnly.jsx
// ...other code
<div className={"border-b my-2 py-2"}>
<div className={"text-sm text-gray-600"}>#{props.project.code}</div>
<div className={"text-base"}>{props.project.name}</div>
</div>
//...
How would someone even achieve this? The solution itself reside in AST parsers. In the Railway blog mentioned above they used codemod.js and wrote this codemod to replace most of their twin macro cases into simpler ones. So I started digging around and understanding their codemod. I even wrote to Alberto to understand what their html object replacing the `tw.` element was.
Their idea was great I think it made their change a tad easier to deal with. They could incrementally go towards any other solution after replacing the twin macro object with theirs since they weren’t bound to the library anymore. But I wanted to go further and I actually wanted to remove all that boiler plate code on our end. I wanted to remove the object declaration of twin macro and have inline html element instead. So I wrote a codemod !
Using babel to remove babel was paradoxal but fun to deal with.
The codemod that did most if it
// This is not optimized code it just works.
const generate = require('@babel/generator').default;
const parse = require('@babel/parser').parse;
const traverse = require('@babel/traverse').default;
const t = require('@babel/types');
const fs = require('fs');
const path = require('path');
function processFile(filePath) {
const stats = fs.statSync(filePath);
if (
stats.isFile() &&
path.extname(filePath) === '.tsx' &&
!path.basename(filePath).includes('stories')
) {
console.log(`Transforming file: ${filePath}`);
const yourSourceCode = fs.readFileSync(filePath, 'utf-8');
if (!yourSourceCode.includes('tw')) {
console.log(`Skipping file (found "styled"): ${filePath}`);
return;
}
const ast = parse(yourSourceCode, {
sourceType: 'module',
plugins: ['jsx', 'typescript'],
});
const twComponents = new Map();
// Collect tw components
traverse(ast, {
VariableDeclarator(path) {
if (path.node.init && t.isTaggedTemplateExpression(path.node.init)) {
const tagged = path.node.init;
// Handling the tw.input case
if (
t.isMemberExpression(tagged.tag) &&
tagged.tag.object.name === 'tw'
) {
const htmlTag = tagged.tag.property.name;
const tailwindClasses = tagged.quasi.quasis[0].value.raw
.replace(/\s+/g, ' ')
.trim();
twComponents.set(path.node.id.name, { htmlTag, tailwindClasses });
}
// Handling the tw(Input) case
else if (
t.isCallExpression(tagged.tag) &&
t.isIdentifier(tagged.tag.callee) &&
tagged.tag.callee.name === 'tw'
) {
const componentName = t.isIdentifier(tagged.tag.arguments[0])
? tagged.tag.arguments[0].name
: 'div';
const tailwindClasses = tagged.quasi.quasis[0].value.raw
.replace(/\s+/g, ' ')
.trim();
twComponents.set(path.node.id.name, {
htmlTag: componentName,
tailwindClasses,
});
}
}
},
});
// Replace JSX Elements using tw components
traverse(ast, {
JSXElement(path) {
if (
t.isJSXIdentifier(path.node.openingElement.name) &&
twComponents.has(path.node.openingElement.name.name)
) {
const { htmlTag, tailwindClasses } = twComponents.get(
path.node.openingElement.name.name
);
path.node.openingElement.name = t.jsxIdentifier(htmlTag);
if (path.node.closingElement) {
path.node.closingElement.name = t.jsxIdentifier(htmlTag);
}
path.node.openingElement.attributes.push(
t.jsxAttribute(
t.jsxIdentifier('className'),
t.stringLiteral(tailwindClasses)
)
);
}
},
// Remove import of tw if no styled found
ImportDeclaration(path) {
if (
path.node.source.value === 'twin.macro' &&
!yourSourceCode.includes('styled')
) {
path.remove();
}
},
});
// Remove tw component declarations for tw.input pattern
traverse(ast, {
VariableDeclarator(path) {
if (path.node.init && t.isTaggedTemplateExpression(path.node.init)) {
const tagged = path.node.init;
if (
t.isMemberExpression(tagged.tag) &&
tagged.tag.object.name === 'tw'
) {
path.remove(); // Remove this variable declarator
}
}
},
});
// Remove tw component declarations for tw(Input) pattern
traverse(ast, {
VariableDeclarator(path) {
if (path.node.init && t.isTaggedTemplateExpression(path.node.init)) {
const tagged = path.node.init;
if (
t.isCallExpression(tagged.tag) &&
t.isIdentifier(tagged.tag.callee) &&
tagged.tag.callee.name === 'tw'
) {
path.remove(); // Remove this variable declarator
}
}
},
});
const transformedCode = generate(ast, { retainLines: true }).code;
fs.writeFileSync(filePath, transformedCode);
console.log(`File transformed successfully: ${filePath}`);
}
}
function traverseDirectory(dirPath) {
const files = fs.readdirSync(dirPath);
files.forEach((file) => {
const filePath = path.join(dirPath, file);
const stats = fs.statSync(filePath);
if (stats.isDirectory()) {
traverseDirectory(filePath); // Recursively call for nested folders
} else if (stats.isFile() && path.extname(filePath) === '.tsx') {
processFile(filePath);
}
});
}
const inputPath = process.argv[2];
if (!inputPath) {
console.error('Please provide a file or directory path.');
process.exit(1);
}
const stats = fs.statSync(inputPath);
if (stats.isDirectory()) {
traverseDirectory(inputPath); // Handle directory
} else if (stats.isFile()) {
processFile(inputPath); // Handle individual file
}This codemod was it, it did almost 70% of the work, took me about a day to write and was replacing all inline declaration of twin macro object it could find. Our codebase was structured in an atomic way using Brad Frost atomic design technique and to say the least this helped me a lot, I could incrementally change our codebase by targeting just subsets of components/folders. Of course it didn’t do all the edge case but then we only had to chip in and do around 30% by hands. Which is still a lot but it’s 70% less than the original idea. The codemod was a great approach it removed the imports, removed the declaration, replaced the inline twin macro component into html element and inlined tailwind strings into className strings.
On the plus side of things, doing part of it by hand also made us realized we could improve components, remove some unused classes, remove unused components and integrate tailwind-merge to handle some edge case.
The top of the mountain
Overall this change was more than welcomed, It took less than 12 days to finish the whole thing. As a solo dev it was quite an accomplishment. Between migrating all other components by hand, fixing small issue in prod, fixing some three shaking import that were discovered during this migration and making sure the app was still running smoothly; I would say it was definitely worth it. But in the end what mattered most is the amount of code we fixed and removed that we didn’t even knew about. Removing that technical debt was so satisfying.
After all this, the performance improvement we saw on day one was staggering. We went from an average of 20 seconds cold start without cache to an average of 10 seconds. We also saw improvement of the same magnitude on HMR/fast reload and the best of all: i’ve never heard my colleague talk about local performance since then. That ~50% improvement on multiple aspect of local development was indeed very worth it. So all and all the mountain was not too high and it was a successful migration. We ended up removing babel, twin macro and styled components all together. There is still work to be done in other aspect and parts of the app but let’s keep it for later.
As a bonus every time tailwind releases new improvements we don’t have to wait for twin macro to be updated and have directly access to it. Like the upcoming release of the promising Tailwind 4.0.
Special thank to Alberto and the Railway Blog.