Choose timezone
Your profile timezone:
AI for Operations: Agents
→
US/Central
Meeting assets for Hardware-Aware AI: Next-Generation Data Quality Monitoring are ready!
Meeting summary
Quick recap
The meeting focused on Agentic Systems for detector operations, with presentations on Archie and knowledge extraction in the context of Genesis IDAs. Nesar Ramachandra described the evolution of Agentic systems, highlighting the integration of LLMs with scientific tools and infrastructure, the use of MCP servers, and the ability to run jobs on HPCs with guardrails for safety. Jason from the Archie team explained how Archie consolidates scattered, heterogeneous experiment knowledge into a shared knowledge graph, detailing its architecture, privacy model, and recent performance improvements over previous vector database approaches. Juan Pablo presented the current infrastructure deployments of Archie for CMS computing operations and WizDQM, comparing virtual machine and Kubernetes setups and discussing their operational differences. The discussion covered technical details such as data ingestion, graph building, metadata extraction, and user feedback mechanisms. Participants asked about the pros and cons of local versus remote data access, safety measures, model hosting, and the practical use of feedback for system improvement. The conversation ended with a mention of an upcoming workshop to further discuss these tools and collaborations across experiments.
Next steps
Jason
- Look into the behavior of Indico ingestion, specifically whether links in Indico slides/PDFs are automatically extracted and added as nodes to the knowledge graph.
- Provide more technical details offline about the knowledge graph ingestion and maintenance process.
- Investigate how to automatically maintain and build knowledge graphs to minimize the user's burden.
Juan Pablo
- Add Juan Pablo to the relevant Slack channel for follow-up discussions.
- Continue working towards full automation of the Helm chart deployment process for Archie, including adopting GitOps for automatic upgrades.
- Work on centralizing self-hosted MCP servers and managing them through an SSO gateway.
Nesar
- Upload the slides and provide tutorial links for building and using Agentic systems, including how to build custom servers and clients.
Samantha
- Incorporate user feedback (thumbs up/down, comments) from Archie into evaluations or benchmarking to improve the system.
- Further expand the use of metadata to inform the agent's behavior for different shift types or subsystems in WISDQM.
Walter
- Forward the link to the Archie workshop (October 19-23) to interested parties, including Nesar, Azton, and Jenny.
Summary
Agentic Systems Knowledge Extraction Overview
The meeting focused on Agentic Systems, specifically Archie and knowledge extraction for detector operations. Nesar Ramachandra from Argonne National Laboratory presented an overview of Agentic Systems evolution, explaining how they have transitioned from first-generation systems with limited tool access to current second-generation systems that can interface with thousands of tools and run on HPCs. He demonstrated how Agentic Systems can handle complex scientific workflows, including accessing and processing data from multiple sources and running jobs on supercomputers.Jason from MIT presented the Archie system, which creates a structured knowledge base from scattered organizational data sources like JIRA tickets, documentation, and operational databases. He shared results from production deployment at CMS CompOps, showing that the knowledge graph backend provided a 13% quality improvement over the previous vector database approach. The system includes privacy controls through anonymization and supports incremental data ingestion.The discussion revealed that operations will not make remote HPC calls during operations, focusing instead on local processing. Participants explored the potential for using Genesis mission infrastructure for future operations, including AmSC services and data lakehouses.
Knowledge Graph Maintenance Techniques
Jason explained how knowledge graphs are built and maintained in his system, describing the process of defining data sources, extracting information, and storing the graph in a Postgres database. He noted that while knowledge graphs offer advantages over simpler approaches like grep, the overhead of maintaining them can be a challenge, which is why his research focuses on automatic maintenance techniques. When asked about MCP interface for Archie, Jason clarified that Archie itself functions as a chat interface with an MCP server and knowledge base, making it more practical for agents to connect directly to the knowledge base rather than through Archie.
Archie Deployment and Infrastructure Updates
Juan Pablo presented two current deployments of Archie, focusing on the CMS COMPOPS and WISDQM instances. He explained the differences in infrastructure, upgrade processes, and scalability between the virtual machine and Kubernetes deployments. Jason answered questions about relationship extraction in Archie, explaining that it primarily uses deterministic extraction through Python functions and metadata. The team discussed the potential for automatically following JIRA links in Indico slides, with Jason agreeing to investigate this further. The group also touched on the upcoming Atlas search workshop, where Jason will have the opportunity to discuss similarities and potential collaboration with the Atlas team.
There are minutes attached to this event.
Show them.