Our PIM team convened for a tech hackathon centered around AI. Over two days, we collaborated to build prototypes leveraging OpenAI's Assistants API. Here are the key insights I've gathered from organizing and participating in the event.
At the end of January, our PIM (product information management) team held a joint tech meeting (aka hackathon) dedicated to AI. The format was the following:
Collecting ideas upfront.
Building teams around ideas.
Building prototypes during the two-day event.
After some sorting and shuffling, two teams (six people per team) were formed to work on these ideas:
AI-powered product catalog setup.
AI-powered MC navigation.
Both ideas relied upon the new Assistants API by OpenAI that allows customizing ChatGPT into specialized assistants and using them via API.
Long story short — the event went well. Both teams managed to build prototypes, we had fun and the feedback afterward was very positive. Here are the lessons I learned from organizing and participating in the event.
The new OpenAI developer platform showed some quirks. There were inconsistencies in behavior between different users (e.g. for some of them the GPT-4 model was available without payment while this wasn't the case for others) and just plain “500” errors popped up several times. The platform is still in Beta, so that's okay. But I take it as another reminder that Beta is Beta regardless of the prominence of the name behind the product.
When developing software is your daily work, you might forget how much effort and intelligence is behind the modern development experience. IDEs, best practices, patterns, etc. — when we were crafting non-trivial prompts collaboratively, I was reminded how it feels to pack a piece of logic into a wall of text. Reading a prompt is not like reading a JavaScript file. I felt like spending much more cognitive power to grasp the logic written in natural text compared to the pieces of code. I’m not even touching modularity.
“Natural language is the new programming language,” is a bold statement people like to make nowadays and, sure enough, there is sense to it. But, there's definitely a long way to go.
We know that ChatGPT can answer the same question differently. But, whenever I used it before for one-shot jobs, I never had an issue with it. However, once someone starts embedding it in automated workflows, that variance becomes much more obvious and problematic. There are tools to deal with it, but one definitely should expect to spend some time taming it.
When planning the event, I included multiple sessions with the advisors in two days. That appeared to be a two-sided coin. While the feedback about the helpfulness of advisors was definitely very positive, there was also feedback about too many meetings subtracting from the teamwork time.
So, next time I would either keep it smaller (combining consulting multiple advisers into one session) or more flexible (asynchronous ad-hoc questions instead of fixed meetings).
While working in a remote-first team with well established practices, it’s easy to forget about the time spent to get there. This was a reminder. When quickly starting the project with a new team in a remote setting, some initial inefficiencies should be included into the “overhead budget.”
One of the most popular messages in the feedback I received was, “An additional day would be good.” Overall, a three-day event is more appropriate for such an occasion.
We used the paid services for OpenAI and set up our accounts upfront. But, I wish I had invested more effort in arranging team billing. Dealing with individual bills before, during and after the event was quite an effort.
Pro tip: Don’t ask people to delete accounts before they've exported the receipts.
Since the event had a manageable scale, I wore both organizer’s and participant’s hats.
I expected some pitfalls there, so unsurprisingly, I made some mistakes during the event — like overseeing an inquiry from another team about expired API credits.
Thus, I found that a hackathon of the scale that was organized should be the size limit for one person sharing organizer and participant roles. Going bigger requires a dedicated organizer during the event.
Want to learn more about how commercetools is leveraging AI assistants? Then check out commercetools Foundry.
Ivan Panov is an Engineering Manager at commercetools. He has over 15 years of experience in software engineering, including multiple leadership roles. His background is varied, encompassing small startups, large corporations, product-based companies, traditional industries, and outsourcing firms across multiple domains such as healthcare and eCommerce.