Projects
Merged the sort-sales-by-invoice-number branch into main and deployed to production. Ran the task to update the index mapping for invoice_number and ran into an issue with the normalizer. Since the keyword_sort_normalizer did not already exist in the index's settings, it was not added when I updated the mapping, which caused the task to fail. Removed the normalizer from the mapping and ran the task again successfully. Going to review Dania's work on adding new sales filters today.
Incidents
Sales Not Appearing in Sales History (SEV-1)
The incident involved new sales not displaying in Sales History for some accounts, while others saw delayed or normal behavior. Investigation found that while sales were being recorded, there was significant consumer lag in the sale document producer, particularly on certain Kafka partitions. This lag was correlated with increased load, possibly due to a slow MySQL query running a full table scan behind a feature flag for staff accounts. The team mitigated the issue by increasing the number of consumer instances and disabling the problematic feature flag, which led to a reduction in lag and sales appearing as expected. No critical alerts were triggered for this lag, which has been noted as an area for improvement. The incident was closed as resolved, though some minor front-end discrepancies and a need for further investigation into the root cause and monitoring remain.
Next steps include scheduling a Post-Incident Review to investigate the root cause and document findings, reviewing and improving alerting for consumer lag and slow queries, evaluating and optimizing the problematic MySQL query (potentially with improved indexing), and continuing to monitor Sales History and consumer performance for stability.
Journal
After the makeathon awards ceremony, I was impressed with many of the projects and talent shown by the engineers and teams at the company. While I respect their technical accomplishments, it brought up some concerns I have about the ethical considerations of certain projects and the potential for misuse of the technology.
I think it's important for those of us who are engineers to consider the ethical implications of our work and to ensure that while we push the boundaries of what's possible, we are also being responsible and mindful of the potential consequences. In particular, I think we need to consider a code of ethics around the use of AI in surveillance and data collection. AI has the potential to be a powerful tool for improving our lives and solving complex problems, but it also has the potential to be used in ways that violate people's privacy and civil liberties.
What kinds of safeguards can we put in place to ensure that AI is used responsibly and ethically in these contexts? What kind of oversight and accountability mechanisms can we implement to prevent misuse? We must acknowledge the realities of the world and governments our software operates in and the impact it can have on society.
Just as we have a responsibility to ensure we are not leaking sensitive company data to the AI models we use, we should also have the same responsibility to ensure we are not submitting personal data of unconsenting individuals to the AI models we use, especially when we have no control over what information is collected from surveillance footage or how that data is used or stored by the AI providers. I think it's important for us to have these conversations and to work together to develop ethical guidelines and best practices for the use of AI in surveillance and data collection, to ensure that we are using this technology in a way that benefits society as a whole and respects people's rights and freedoms.