I celebrated my 1-year work anniversary at ASAPP this past week. So now that I possess one year of professional experience as a machine learning engineer (aka ML engineer or MLE) – what has this past year been like for me?

Before ML, I spent 5 years as a traditional software engineer, building web services and web-based technology for Microsoft OneNote. I love the web, and I loved the new-initiative energy surrounding the Services team in particular at OneNote (an Office product that existed for a solid 10 years before this effort to integrate more natively with cloud service architecture). I loved the mission to empower folks to Remember, Think, and Organize Better – through note-taking and related augmentations.

To review this first year as an MLE, I used Julia Evans’ excellent brag doc template. One thing that I learned from leaving Microsoft in 2018 - it takes a lot of effort to reflect back on five years of work all at once in order to update a resume… so let’s not leave all reflection until the next job hunt. I liked that this chronicling encouraged both detailed recollection and big picture reflection. It inspired me to go a step further and produce a thematic summary for this blog post!

The major theme of 2019 for me was transition, and the career transitions came in several forms: from big tech to a startup, from a consumer-oriented product group to an enterprise company, from the west coast and its Silicon Valley influence to New York City, from individual contributor to tech lead, from web services to… actually, I’m still writing web services (just with ML models baked into them).

Startup

This year was uncomfortable. But discomfort and growth go hand in hand.

My main discomfort stemmed from the upsurge in individual responsibility required in my role. I left Microsoft with the goal in mind to experience a radically different company setup and culture, and I got it. Being at a startup has pushed me to adapt and adapt quickly. The startup mentality is pressurized iteration, imperfection, risk, uncertainty, and some spin (true of most companies). Pivots and shifting goalposts. Big swings and misses. I think these are all well-known enough points to border on startup cliches. I’ve experienced it all here already. I’ve had to learn to push through the discomfort and update my personal desires for consistency, certainty, and unattainable perfection. I’m also learning how and when to use these same instincts to contribute uniquely to my team, tweaking my desire for consistency to a push for system stability, my desire for certainty to deliberate design, my desire for perfection to quality and purposeful organization.

My new company has also bucked and defied unhealthy startup cliches: there is an awareness and active avoidance of burnout (it still happens, but more at an individual level, rather than company-wide); acknowledgement of time spent beyond the regular workweek as extraordinary (there was a crunch time in ~Mar 2019 that required weekend work that the company provided PTO time later to cover); largely egoless and considerate communication (see the recent Away scandal for the exact opposite culture); the satisfaction of selling a real, useful product to real customers.

It can be fairly argued that at 300+ employees, ASAPP is no longer a startup. As we focus on scale and face inevitable growing pains, this will become even more true. Maybe this is the last year I’ll say that I work at a startup. I think what I said above still applies to my experience though, especially given the enormous size of my previous employer.

Enterprise

To be honest, enterprise was a dirty word in my mind before joining ASAPP. All the most boring work at my old job seemed to have “enterprise solutions” at its core. All the most icky and convoluted code bases too (case in point: SharePoint). So when considering this move to the company, its enterprise focus gave me some pause.

Enterprise at ASAPP versus enterprise at Microsoft is different in one crucial, saving way for me: I don’t feel nearly as removed from our customers and end users at ASAPP. Careful and specific relationship management is so key to our company’s success that we have to get into the weeds and know our partners in order to integrate deeply.

There are still downsides to enterprise, like what I’ll call The Most Important Deal Ever syndrome - every deal and every partner does and should matter, but it was somewhat disillusioning in my first few months to be smacked with enough consistent urgency to start wondering if anything is really that urgent (this has settled down more recently, thankfully). There’s still a bit of disconnect for me on who are (or who should be) our primary customers – the business or the people relying on the business (whether the people chatting in or the agents responding). And balancing tensions between these customer tiers on occasion.

New York City

DIVERSITY in surroundings. Sadly not reflected at the office.

I love being able to live in a neighborhood like Bedford-Stuyvesant, Brooklyn as a black woman. San Francisco and Seattle could never and had nothing like Bed-Stuy. I love not sticking out everywhere I go, as was often the case during my time on the west coast.

People are doing a lot of different things in this city too. I have friends in social work, law, medicine, and small business owners (who don’t claim to be a “startup”). I craved this melting pot of professions on the west coast.

I also appreciate the professional diversity for my own career ambitions - in that I do see myself eventually migrating towards more interdisciplinary companies, with cultures and missions that are much less tech-centric.

Tech Lead

Seeing myself as a “technical leader” has probably been my most difficult area of personal growth. Startup life has expanded and rapidly evolved my individual responsibility at work (yay growth, but also scary growth).

I have been learning about myself, my preconceived notions of what leadership looks like, my strengths and how to apply them. I got lucky in having a fantastic female manager who knows how to push me both technically and interpersonally.

My favorite new phrase (from our CEO) is “radical ownership” - I’ve learned to embrace taking on and involve myself in things that aren’t necessarily my designated responsibility, for the good of my team and company.

MLE is a fairly small team (15-20 members) with plenty of opportunity for high impact without too much bureaucracy/barrier to entry. I’m learning to rise to the occasion and play big.

Web Services + ML

In my experience so far, writing ML code is much less like Software 2.0, much more like writing “1.0” code that can tame and coax the output of neural nets into behavior that is consistent, reliable, and reasonable for the business.

This falling short of 2.0 expectations exhibits itself in the fallacy and hidden complexities of something a manager on my team likes to say - “we don’t INC.” This is the idea that the MLE team has never caused a production incident. This isn’t exactly true - it’s just that ML experiences these “soft failures” that appear more permissible because they are harder to track down and control. The range of behaviors our systems can produce is so large that if some fraction of these behaviors are a little odd but not considered borked enough to flag as truly broken, we can just keep rolling. It’s part of the point of ML not to explicitly control these behaviors, until something in the business forces us to… and then we known good.

My first shock in joining the MLE team was our “known goods” system. My first assignment was to fix it up. It is essentially a system for hardcoding rules for system behavior that would be embarrassing to get wrong.

I’m being a bit cynical, in that our team does achieve real machine learning successes at scale. But ML is still a lot of Oz hiding behind the curtain wizardry, a lot of innovative engineering and hackery. I think it is good to talk about it as it is.

2020

So - there it is. I am a machine learning engineer now, primarily concerned with building services containing a machine learning core.

I spent 2019 familiarizing and acclimatizing with all of my transitions (including several personal ones not discussed here). All of these changes at once had me feeling very close to drowning at times, but I’m so glad to have made it this far. So with a bit more establishment and grounding now in 2020, one of my goals is to start turning outwards again by distilling my on-the-job learnings in a generally applicable way through this blog.

Opinions expressed are solely my own and do not express the views or opinions of my employer.