I spent the last two days at Colab26, a hackathon in the Maldives, building RTI4All, an agentic AI system for processing Right to Information (RTI) requests at the Ministry of Climate Change, Environment and Energy. The project was selected as a Wave 1 pilot through the CoLab National AI Use Case Selection Sprint.
This is a technical writeup covering the architecture, the dual-layer retrieval system, the completeness scoring mechanism, and the performance impact of rewriting from Python to Go.
Problem Context
The Maldives Right to Information Act (Act No. 1/2014) requires government ministries to respond to citizen information requests within 21 days with possible extension up to additional 14 days. The operational bottleneck is that officers manually draft every response, often citing the same precedents repeatedly. Citizens submit vague or duplicate requests because they cannot see what has already been answered.
RTI4All automates the retrieval and drafting steps while keeping the officer in the approval loop. Every AI-generated response requires human review before publication.
Architecture: 5-Stage Agentic Workflow
The system implements a multi-stage workflow with explicit quality gates:
┌──────────────────────────────────────────────────────────────────────┐
│ ① Structurer agent (Claude Sonnet, JSON-only) │
│ Subject + description → JSON analysis │
│ │
│ { request_type, key_questions, information_sought, │
│ time_period, geographic_scope, urgency_indicators, │
│ completeness_score ∈ [0,1], │
│ missing_information, related_policies, │
│ estimated_complexity, suggested_response_approach, │
│ relevant_precedents } │
└──────────────────────────────┬───────────────────────────────────────┘
│
completeness_score < 0.7?
┌────────────────┴────────────────┐
YES │ │ NO
▼ ▼
┌────────────────────────┐ ┌──────────────────────────┐
│ ② Officer review queue │ │ ② Retriever │
│ status="Under Review"│ │ - TF-IDF top-4 │
│ (no draft yet) │ │ - Token-graph top-7 │
│ │ │ - Dedupe by id │
│ Officer reads JSON │ │ │
│ analysis, asks for │ │ ↓ │
│ clarification, or │ │ ③ Drafter agent │
│ writes draft manually │ │ (Claude Haiku) │
└────────────────────────┘ │ System prompt = │
│ archive context │
│ "Cite RTI ids, plain │
│ prose, 4-8 sentences" │
└────────────┬─────────────┘
│
▼
┌──────────────────────────┐
│ status="Under Review", │
│ response = draft text │
│ (still needs human │
│ approval before publish)│
└────────────┬─────────────┘
│
▼
┌──────────────────────────┐
│ ④ Officer decision │
│ • Approve → Responded │
│ • Reject → Rejected │
│ • Clarify → Request │
│ Clarification │
└────────────┬─────────────┘
│
Approve path │
▼
┌──────────────────────────┐
│ ⑤ Feedback loop │
│ rag.IndexResponded(req)│
│ graph.UpdateForRequest │
│ → next draft can cite │
│ THIS one │
└──────────────────────────┘
The critical design decision is gating on completeness score before drafting. Requests scoring below 0.7 skip the drafting stage entirely and go straight to officer review with only the structured analysis. This prevents officers from reviewing low-quality AI outputs generated from ambiguous inputs.
Stage 1: Request Structuring with Claude Sonnet
The Structurer agent uses Claude 3.5 Sonnet to parse free-text requests into a JSON schema with 12 fields, including a completeness_score from 0 to 1.
func (c *Client) ProcessRequestStructure(ctx context.Context, subject, description, departmentID string) map[string]any {
if !c.Configured() {
log.Println("[ai] no API key — using fallback request structure")
return fallbackStructure(subject, description)
}
prompt := fmt.Sprintf(`You are an AI assistant helping process Right to Information (RTI) requests in the Maldives.
Analyze the following RTI request and provide a structured analysis in JSON format.
CITIZEN REQUEST:
Subject: %s
Description: %s
Department: %s
Provide your analysis as a JSON object with these fields:
1. "request_type": Classify as "Data Request", "Policy Clarification", "Document Access", "Budget Information", "Procedure Inquiry", or "Other"
2. "key_questions": List 2-4 main questions the citizen is asking
3. "information_sought": List specific data, documents, or information items requested
4. "time_period": Extract any time period mentioned or null
5. "geographic_scope": Extract any geographic scope or null
6. "urgency_indicators": List any time-sensitive aspects (empty array if none)
7. "completeness_score": Rate 0.0 to 1.0 how complete and clear the request is
8. "missing_information": List what additional information would make this request clearer (empty array if complete)
9. "related_policies": List relevant Maldivian laws, policies, or RTI Act provisions (empty array if none obvious)
10. "estimated_complexity": "Simple", "Moderate", or "Complex"
11. "suggested_response_approach": Brief 2-3 sentence suggestion on how the officer should approach responding
12. "relevant_precedents": List any similar types of requests that might have been processed before (empty array if none)
Respond ONLY with the JSON object, no other text.`, subject, description, departmentID)
model := os.Getenv("ANTHROPIC_STRUCTURE_MODEL")
if model == "" {
model = "claude-3-5-sonnet-20241022"
}
raw, err := c.callMessages(ctx, apiMessageRequest{
Model: model,
MaxTokens: 2000,
Messages: []apiMessage{{Role: "user", Content: prompt}},
})
// ... error handling and JSON parsing
}
Sonnet was chosen for this task because its structured output reliability is high. In testing on sample requests, it returned valid JSON on every invocation, whereas Haiku occasionally produced malformed output when handling the 12-field schema. The Structurer runs on every incoming request, so robustness matters more than latency.
The completeness_score is derived by having the model evaluate the request against a checklist: subject matter clarity, scope definition, time period, format preferences, and specific data points. Testing on 20 sample requests showed the model's scores aligned with manual human evaluation.
Stage 2: Dual-Layer Retrieval
Requests with completeness score ≥ 0.7 proceed to retrieval. The system uses two complementary methods to find relevant context from the ministry archive.
Layer 1: TF-IDF Cosine Similarity
The first retrieval layer is classical information retrieval: TF-IDF with cosine similarity.
// Package rag implements a small in-process retrieval layer used to ground
// the AI's drafts in the ministry archive (past responded RTI requests +
// FAQs).
//
// The original Python implementation used sentence-transformers
// (all-MiniLM-L6-v2) for embeddings. Pulling 300 MB of PyTorch into a Go
// binary is impractical, so this Go port uses a deterministic TF-IDF +
// cosine-similarity retriever instead. The retrieval contract is identical:
// callers supply a query string and get back ranked payloads with a `_score`
// in [0, 1].
func tokenize(text string) []string {
matches := tokenRe.FindAllString(strings.ToLower(text), -1)
out := make([]string, 0, len(matches))
for _, m := range matches {
if len(m) < 3 {
continue
}
if _, skip := stopWords[m]; skip {
continue
}
out = append(out, m)
}
return out
}
func (i *Index) recompute(rawDocs []map[string]int) {
// Recompute IDF and per-doc TF-IDF vectors from raw term frequencies.
i.docFreq = map[string]int{}
for _, tf := range rawDocs {
for term := range tf {
i.docFreq[term]++
}
}
i.idf = map[string]float64{}
n := float64(len(rawDocs))
for term, df := range i.docFreq {
// Smoothed inverse document frequency.
i.idf[term] = math.Log((n+1.0)/(float64(df)+1.0)) + 1.0
}
i.docCount = len(rawDocs)
i.vectors = make([]docVector, len(rawDocs))
for idx, tf := range rawDocs {
i.vectors[idx] = i.buildVector(tf)
}
}
The retrieval is in-process, zero-dependency, and boots in microseconds. No model download, no GPU, no Python interpreter. For government RTI text, which is keyword-heavy (place names, statute numbers, programme names), TF-IDF gives competitive results with embedding models while being orders of magnitude faster for in-process inference.
The implementation is read-locked for concurrency, allowing multiple requests to query the index simultaneously while still supporting atomic updates when an officer approves a response.
func (i *Index) Retrieve(query string, k int) []Hit {
i.mu.RLock()
defer i.mu.RUnlock()
if len(i.ids) == 0 || strings.TrimSpace(query) == "" {
return nil
}
qTF := map[string]int{}
for _, tok := range tokenize(query) {
qTF[tok]++
}
qVec := i.buildVector(qTF)
if len(qVec.terms) == 0 {
return nil
}
type scored struct {
idx int
score float64
}
scores := make([]scored, 0, len(i.vectors))
for idx, v := range i.vectors {
s := cosine(qVec, v)
if s > 0 {
scores = append(scores, scored{idx, s})
}
}
sort.Slice(scores, func(a, b int) bool { return scores[a].score > scores[b].score })
if k > len(scores) {
k = len(scores)
}
out := make([]Hit, 0, k)
for _, s := range scores[:k] {
payload := make(map[string]any, len(i.payloads[s.idx])+1)
for k, v := range i.payloads[s.idx] {
payload[k] = v
}
out = append(out, Hit{Payload: payload, Score: s.score})
}
return out
}
Layer 2: Token Co-occurrence Graph
Pure keyword matching misses semantically related precedents. If a citizen asks about "reef monitoring data" but a past request used "coral surveys", the TF-IDF retriever will not connect them.
The graph layer solves this by building a token co-occurrence graph at startup and expanding queries by one hop.
// Package graph is a lightweight graph-augmented retrieval layer over the
// ministry archive.
//
// The original Python implementation shelled out to the `graphify` CLI to
// build an LLM-extracted entity graph. That tool has no Go counterpart and
// would balloon the container image, so this port uses a deterministic
// token-cooccurrence graph instead:
//
// - Nodes are notable lowercase tokens (≥ 4 chars, non-stopwords) that
// appear in at least one corpus document.
// - Two nodes share an edge if they co-occur in the same document.
// - Retrieval matches the query tokens against node labels, traverses
// one hop of edges, and ranks the documents whose tokens were touched.
//
// Empirically this gives "graph-linked precedent" retrieval for the small
// ministry archive: shared concepts in a query surface other requests that
// mention the same concepts, even when no individual term overlaps exactly.
func (s *State) addDocLocked(id string, payload map[string]any, text string) {
terms := uniqueTerms(text)
idx := len(s.ids)
s.ids = append(s.ids, id)
s.docs = append(s.docs, payload)
s.docTerms = append(s.docTerms, terms)
for t := range terms {
s.termDocs[t] = append(s.termDocs[t], idx)
}
// Update co-occurrence counts between every pair of terms in this doc.
for a := range terms {
row, ok := s.cooccur[a]
if !ok {
row = map[string]int{}
s.cooccur[a] = row
}
for b := range terms {
if a == b {
continue
}
row[b]++
}
}
}
At retrieval time, query terms are expanded by taking the top-8 co-occurring neighbors from the graph:
func (s *State) Retrieve(query string, k int) []rag.Hit {
s.mu.RLock()
defer s.mu.RUnlock()
if len(s.ids) == 0 || strings.TrimSpace(query) == "" {
return nil
}
qTerms := uniqueTerms(query)
if len(qTerms) == 0 {
return nil
}
// Expand the query terms by one hop along the cooccurrence graph,
// inheriting a decayed weight to favour direct matches.
weights := map[string]float64{}
for t := range qTerms {
weights[t] = 1.0
}
for t := range qTerms {
row := s.cooccur[t]
// Take the top-N neighbours by cooccurrence count to keep the
// expansion focused on the strongest links.
type pair struct {
term string
count int
}
neighbours := make([]pair, 0, len(row))
for nt, c := range row {
neighbours = append(neighbours, pair{nt, c})
}
sort.Slice(neighbours, func(i, j int) bool { return neighbours[i].count > neighbours[j].count })
max := 8
if len(neighbours) < max {
max = len(neighbours)
}
for _, n := range neighbours[:max] {
if _, seen := weights[n.term]; seen {
continue
}
weights[n.term] = 0.5
}
}
// Score every document by the sum of weights for its terms.
type scored struct {
idx int
score float64
}
scores := make([]scored, 0, len(s.ids))
for idx, terms := range s.docTerms {
var sum float64
for t := range terms {
if w, ok := weights[t]; ok {
sum += w
}
}
if sum > 0 {
scores = append(scores, scored{idx, sum})
}
}
if len(scores) == 0 {
return nil
}
sort.Slice(scores, func(a, b int) bool { return scores[a].score > scores[b].score })
if k > len(scores) {
k = len(scores)
}
out := make([]rag.Hit, 0, k)
for _, sc := range scores[:k] {
payload := make(map[string]any, len(s.docs[sc.idx]))
for k, v := range s.docs[sc.idx] {
payload[k] = v
}
out = append(out, rag.Hit{Payload: payload, Score: sc.score})
}
return out
}
This replaces the graphify CLI from the Python implementation. The original design called out to a separate process that used an LLM to extract entities and relations. That approach required shipping a 1 GB Docker image and added 500-800ms of latency per request. The deterministic graph builds in 10-15ms at startup for a 100-document corpus and runs with zero marginal cost per query.
Stage 3: Grounded Drafting with Claude Haiku
Both retrieval layers feed into the Drafter agent's system prompt. The Drafter uses Claude Haiku 4.5, which is optimized for template-constrained summarization.
const answerSystemTemplate = `You are an AI assistant for the Maldives Ministry of Climate Change, Environment and Energy's citizen Right to Information (RTI) portal.
When a citizen submits an RTI request, your job is to draft a clear, factual response addressed to the citizen, grounded in the ministry's local archive of past responded RTI requests and standing FAQs.
The archive is retrieved for you two ways and shown below:
- Vector matches: items semantically similar to the question.
- Graph-linked items: items that share key concepts with the question.
These are authoritative precedent and process knowledge. PREFER them when they answer the question.
If the archive does not contain enough information to answer fully, say so plainly and direct the citizen to file a formal follow-up RTI application with the Information Officer specifying the missing detail.
MINISTRY ARCHIVE — VECTOR MATCHES:
%s
MINISTRY ARCHIVE — GRAPH-LINKED PRECEDENT:
%s
OFFICIAL SOURCES THE CITIZEN MAY ALSO CONSULT:
- rtidhonbe.com — the public Maldives RTI vault (decisions, precedents, published responses).
- environment.gov.mv — the Ministry of Climate Change, Environment and Energy's official site (policies, reports, press releases).
RULES:
- Every factual claim must come from the archive shown above. Do not invent figures, names, dates, or document references.
- Cite the relevant prior RTI id (e.g. RTI-2024-0001) or FAQ id when you draw on it.
- Address the citizen directly. Be concise: 4-8 sentences, plain prose, no markdown headings.
- If your answer can be backed up by a public document on rtidhonbe.com or environment.gov.mv, end the reply with a short "Useful resources:" block listing 1-3 plain URLs (one per line). Only include this block when the resources are clearly relevant; otherwise omit it.
- Do NOT fabricate deep links to specific PDFs you have not actually retrieved — link the section landing pages instead, e.g. https://rtidhonbe.com or https://environment.gov.mv .
OUTPUT FORMAT:
Your reply will be shown to the citizen verbatim. Do NOT include any preamble or signoff like "Response to Citizen:", "Dear Citizen,", or "Best regards". Output ONLY the body of the response, in plain prose, starting with the substantive answer. If you add a "Useful resources:" block, put it on its own paragraph at the end.`
The Drafter agent retrieves top-4 from TF-IDF and top-3 from the graph, deduplicates by ID, and interpolates both blocks into the system prompt. The user prompt is minimal:
func (c *Client) AnswerRequest(ctx context.Context, subject, description string, ragIdx *rag.Index, graphState *graph.State) (string, error) {
if !c.Configured() {
log.Println("[ai] ANTHROPIC_API_KEY not set; returning stub response")
return "[AI service not configured: set ANTHROPIC_API_KEY] Your request to the Ministry of Climate Change, Environment and Energy has been received and is pending review.", nil
}
query := strings.TrimSpace(subject + "\n" + description)
const ragK = 4
const graphK = 3
var vectorHits []rag.Hit
if ragIdx != nil {
vectorHits = ragIdx.Retrieve(query, ragK)
}
var graphHits []rag.Hit
if graphState != nil {
graphHits = graphState.Retrieve(query, graphK+ragK)
}
// Dedupe graph hits against vector hits to keep the prompt compact.
seen := map[string]struct{}{}
for _, h := range vectorHits {
if id, ok := h.Payload["id"].(string); ok {
seen[id] = struct{}{}
}
}
deduped := make([]rag.Hit, 0, len(graphHits))
for _, h := range graphHits {
id, _ := h.Payload["id"].(string)
if _, dup := seen[id]; dup {
continue
}
deduped = append(deduped, h)
if len(deduped) >= graphK {
break
}
}
system := fmt.Sprintf(answerSystemTemplate,
rag.FormatForPrompt(vectorHits),
graph.FormatForPrompt(deduped),
)
userPrompt := fmt.Sprintf("Subject: %s\nDescription: %s\n\nGround your draft in the ministry archive shown in your instructions and write the response to the citizen now.", subject, description)
answer, err := c.callMessages(ctx, apiMessageRequest{
Model: c.Model,
MaxTokens: 1024,
System: system,
Messages: []apiMessage{{Role: "user", Content: userPrompt}},
})
if err != nil {
return "", err
}
if strings.TrimSpace(answer) == "" {
return "", fmt.Errorf("anthropic returned empty answer")
}
return answer, nil
}
The model's entire task is: read the cited evidence, answer in plain prose, cite prior RTI IDs. Haiku was chosen for this stage because it is fast (200-400ms p99 latency versus 800-1200ms for Sonnet), cheap (1/20th the cost per token), and performs identically to Sonnet on tightly constrained summarization tasks where the output format is rigid. The Drafter does not need open-ended reasoning capability; it needs faithful paraphrasing within a strict template.
Stage 4: Officer Review and Feedback Loop
Every drafted response goes to an officer with three actions: approve, reject, or request clarification. When the officer approves, the response is published and immediately indexed back into both retrieval layers:
// After approval, re-index the response into RAG and graph for future retrieval
if updated.Status == "Responded" && updated.Response != "" {
go func() {
ragIdx.Upsert("req:"+updated.ID, updated.Subject+" "+updated.Description+" "+updated.Response, map[string]any{
"kind": "request",
"id": updated.ID,
"subject": updated.Subject,
"description": updated.Description,
"response": updated.Response,
"date_filed": updated.DateFiled,
})
graphState.UpdateForRequest(updated)
}()
}
This creates a feedback loop: each approved response makes the next draft better. The archive grows with the ministry's own precedents, not external training data. The presence of high-quality precedents in the archive also indirectly improves completeness scores for future similar requests, since the Structurer agent can identify relevant precedents in its analysis.
Python to Go Rewrite: Performance Impact
The original implementation was Python (FastAPI, sentence-transformers, graphify CLI). The Go rewrite delivered substantial performance improvements:
| Metric | Python | Go | Improvement |
|---|---|---|---|
| Cold start time | 10-15 seconds | 50 milliseconds | 250x faster |
| Docker image size | 1 GB | 15 MB | 65x smaller |
| Memory at idle | 400 MB | 20 MB | 20x leaner |
| Retrieval latency (p99, TF-IDF query) | 180 ms | 4 ms | 45x faster |
| Embedding model | 300 MB PyTorch | None (TF-IDF) | Removed entirely |
| External subprocess | graphify CLI | None (in-process) | Removed entirely |
The performance benefits stem from three architectural changes:
- Replacing sentence-transformers with TF-IDF. No model loading, no GPU, no Python interpreter startup. The retriever boots in microseconds and runs in-process. For RTI text (keyword-heavy domain), TF-IDF retrieval quality is competitive with embedding-based search while being far faster for local inference.
- Replacing the graphify CLI with a deterministic token graph. The Python version shelled out to an external process that used an LLM to extract entities. The Go version builds a co-occurrence graph deterministically at startup in 10-15ms for a 100-document corpus.
- Switching from Python to Go for the HTTP server. Go's runtime is 20 MB versus Python's 400 MB (including FastAPI, uvicorn, and dependencies). Request handling is synchronous in Go versus async in Python, which simplifies the codebase and eliminates coroutine overhead.
The Go backend compiles to a single static binary with zero runtime dependencies. The Docker image is a multi-stage build that produces a 15 MB artifact:
FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .
FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY --from=builder /app/main .
COPY --from=builder /app/data ./data
EXPOSE 8080
CMD ["./main"]
This makes the system deployable on resource-constrained infrastructure and substantially reduces cloud hosting costs. The entire backend runs comfortably on a 1 vCPU / 512 MB RAM container.
Frontend: TypeScript + React
The frontend is React 18 with TypeScript, Vite, and Tailwind CSS. The production bundle gzips to 70 KB. The TypeScript rewrite (from JavaScript) added compile-time type safety across the entire API surface.
When the backend API schema changed during development, TypeScript caught approximately 15-20 places in the frontend that needed updating. Without static typing, those bugs would have been found one at a time during manual testing. The rewrite cost 2 hours and saved at least 4.
The design system uses a custom grayscale palette with blue accents, optimized for accessibility and mobile responsiveness. The admin panel shows structured request data (completeness score, missing information, estimated complexity) inline with the draft response, which allows officers to make informed decisions without switching contexts.
Data Persistence and Audit Trail
The data model is in-memory with atomic JSON snapshots to disk and rotated backups. Every request has a complete history: when it was filed, when it was analyzed, what the completeness score was, what was retrieved, what the officer did, and when it was published.
State transitions are atomic and logged:
func (db *DB) Save(path string) error {
db.mu.RLock()
defer db.mu.RUnlock()
tempPath := path + ".tmp"
f, err := os.OpenFile(tempPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
if err != nil {
return fmt.Errorf("create temp file: %w", err)
}
enc := json.NewEncoder(f)
enc.SetIndent("", " ")
if err := enc.Encode(db); err != nil {
f.Close()
os.Remove(tempPath)
return fmt.Errorf("encode JSON: %w", err)
}
if err := f.Close(); err != nil {
os.Remove(tempPath)
return fmt.Errorf("close temp file: %w", err)
}
if err := os.Rename(tempPath, path); err != nil {
os.Remove(tempPath)
return fmt.Errorf("atomic replace: %w", err)
}
return nil
}
This approach trades off transactional guarantees for simplicity and portability. For a corpus of hundreds of requests, the entire state fits in memory and writes to disk in under 10ms. If the snapshot is corrupted on boot, the system falls back to the most recent backup.
Known Constraints
The current implementation handles multiple departments (Ministry of Climate Change Environment and Energy, Ministry of Health, Ministry of Education, Ministry of Transport and Civil Aviation, Ministry of Foreign Affairs) with department-specific request routing. Each request is filed to a single department, and the retrieval system searches only that department's past responses and shared FAQs.
Language support. The tokenizer and retrieval layers are English-only. Supporting Dhivehi would require either a multilingual embedding model (reintroducing the 300 MB model size) or a separate Dhivehi-specific TF-IDF index with custom tokenization.
Cross-department requests. Requests that require information from multiple ministries must be handled manually. The retrieval system does not automatically search across department boundaries.
PII and exemption detection. The system does not automatically detect requests that touch personally identifiable information or legal exemptions under the RTI Act. Officers must identify these cases manually during review.
Live data integration. The retrieval layers index past RTI responses and FAQs, but do not search live ministry data assets (budget spreadsheets, policy documents, internal reports). Integrating these would require building connectors to existing document management systems.
Conclusion
RTI4All demonstrates that production-grade agentic AI systems can be built in constrained time frames if the architecture is carefully scoped. The key technical decisions were:
- Explicit gating on completeness score. Ambiguous requests are flagged before drafting, which prevents officer burnout from reviewing low-quality outputs.
- Dual-layer retrieval (TF-IDF + token co-occurrence graph). Combining keyword matching with concept-link expansion gives better recall than either method alone, without requiring embedding models.
- Task-specific model selection. Claude Sonnet for structured JSON parsing with high reliability, Claude Haiku for grounded summarization with low latency. Splitting tasks by model capability optimizes cost and latency.
- Rewriting from Python to Go. Removing the 300 MB sentence-transformers model and the graphify subprocess reduced cold start time from 10 seconds to 50 milliseconds and image size from 1 GB to 15 MB.
- Atomic JSON snapshots with rotated backups. Simple, portable persistence that provides full audit trails without requiring a relational database.
The code is MIT-licensed and available at the project repository. The system is deployable via Docker Compose and requires only an Anthropic API key. The technical implementation is production-ready regardless of competition outcome.