Documentation Index
Fetch the complete documentation index at: https://mintlify.com/tractorjuice/arc-kit/llms.txt
Use this file to discover all available pages before exploring further.
Overview
The /arckit.evaluate command creates objective vendor evaluation frameworks and scores vendor proposals against requirements.
Command Usage
/arckit.evaluate <task description>
Examples
# Create evaluation framework
/arckit.evaluate Create framework for project 001
# Score individual vendor
/arckit.evaluate Score Acme Payment Solutions
# Compare all vendors
/arckit.evaluate Compare all vendors for payment gateway project
Three Evaluation Modes
Create Framework
Score Vendor
Compare Vendors
Generate evaluation criteria before receiving proposalsWhen: After SOW/DOS published, before proposal deadlineOutput: ARC-{PROJECT_ID}-EVAL-v1.0.md
Score a single vendor proposal against frameworkWhen: After receiving vendor proposalOutput: projects/{project}/vendors/{vendor}/evaluation.md
Side-by-side comparison of all vendorsWhen: After scoring all vendors, ready to decideOutput: ARC-{PROJECT_ID}-VEND-v1.0.md
Prerequisites
Mandatory:
- Requirements (
ARC-{PROJECT_ID}-REQ-v1.0.md) - Evaluate vendors against requirements
- Architecture Principles (
ARC-000-PRIN-v1.0.md) - Technology standards, compliance
Recommended:
- Statement of Work (
ARC-{PROJECT_ID}-SOW-v1.0.md) - Pre-defined evaluation criteria
- DOS Requirements (
ARC-{PROJECT_ID}-DOS-v1.0.md) - DOS evaluation approach
- Technology Research (
ARC-{PROJECT_ID}-RSCH-v1.0.md) - Market landscape, vendor options
- G-Cloud Search (
ARC-{PROJECT_ID}-GCLD-v1.0.md) - Shortlisted services
Optional:
- Vendor Proposals in
projects/{project}/vendors/{vendor}/ - For scoring mode
- Stakeholder Analysis (
ARC-{PROJECT_ID}-STKE-v1.0.md) - Evaluation panel composition
Mode 1: Create Evaluation Framework
Mandatory Qualifications (Pass/Fail)
Vendors must meet ALL mandatory qualifications or be disqualified:
Certifications:
- Industry-specific (PCI-DSS for payments, ISO 27001 for security)
- Cloud provider certifications (AWS/Azure/GCP partner status)
- Security clearances (UK public sector: SC, DV)
Experience:
- Minimum years in relevant domain (e.g., 5+ years financial services)
- Similar project references (minimum 2-3)
- Technology stack expertise
Financial Stability:
- Minimum company age
- Professional indemnity insurance
- Financial statements (if required)
References:
- Minimum 2-3 client references from similar projects
- Contactable references
- Recent projects (within 2 years)
Scoring Criteria (100 Points Total)
Standard weightings (customizable):
## Evaluation Criteria (100 Points)
### 1. Technical Approach (35 points)
**Solution Design (15 points):**
- Architecture quality and scalability
- Alignment with architecture principles
- Innovation and best practices
- Risk identification and mitigation
**Technical Compliance (10 points):**
- Coverage of MUST requirements
- Approach to NFR requirements (performance, security, scalability)
- Integration strategy
**Technology Choices (10 points):**
- Appropriate technology stack
- Justification for choices
- Future-proofing and maintainability
### 2. Project Approach (20 points)
**Methodology (8 points):**
- Development methodology (Agile, Waterfall, hybrid)
- Sprint planning and iteration approach
- Stakeholder engagement strategy
**Risk Management (6 points):**
- Risk identification completeness
- Mitigation strategies
- Contingency planning
**Quality Assurance (6 points):**
- Testing strategy (unit, integration, E2E, performance, security)
- Test coverage targets
- QA process and tools
### 3. Team Qualifications (25 points)
**Relevant Experience (12 points):**
- Domain expertise (financial services, healthcare, etc.)
- Technology stack experience
- Similar project delivery
**Team Composition (8 points):**
- Appropriate roles (architects, senior/junior devs, QA, DevOps)
- Team size and availability
- Key personnel CVs
**Certifications (5 points):**
- Professional certifications (AWS Certified, PCI-DSS QSA, etc.)
- Security clearances (if required)
- Training and continuous learning
### 4. Company Experience (10 points)
**Similar Projects (5 points):**
- Number and quality of reference projects
- Relevance to current project
- Client testimonials
**Industry Expertise (3 points):**
- Years in industry
- Market reputation
- Awards and recognition
**Financial Stability (2 points):**
- Company age and growth
- Client retention rate
- Financial health indicators
### 5. Pricing (10 points)
**Cost Competitiveness (5 points):**
- Total cost vs market benchmark
- Value for money assessment
**Pricing Clarity (3 points):**
- Transparent breakdown
- No hidden costs
- Clear payment terms
**Commercial Terms (2 points):**
- Payment milestone alignment
- Warranty terms
- Exit provisions
Example Framework Output
# Vendor Evaluation Framework
**Project**: Payment Gateway Modernization (001)
**Document**: ARC-001-EVAL-v1.0
**Created**: 2026-03-04
## Mandatory Qualifications (Pass/Fail)
Vendors must meet ALL criteria below or be disqualified:
- ✅ **PCI-DSS Certification**: Current QSA certification or 5+ years Level 1 experience
- ✅ **Financial Services Experience**: Minimum 5 years in payment systems
- ✅ **UK References**: Minimum 2 UK financial services references
- ✅ **Cloud Certification**: AWS Certified Solutions Architect (Professional) or equivalent
- ✅ **Team Availability**: Minimum 4 FTE committed for 6-month duration
- ✅ **Proposal Completeness**: All sections completed, no TBD items
## Scoring Criteria (100 Points)
[Full scoring criteria from template above]
## Evaluation Process
**Evaluation Team:**
- Lead Architect (RACI: Accountable)
- Senior Engineer (RACI: Consulted)
- Product Owner (RACI: Consulted)
- Procurement Manager (RACI: Informed)
**Timeline:**
1. Proposal deadline: 2026-03-15
2. Mandatory qualification check: 2026-03-16
3. Individual scoring: 2026-03-17 to 2026-03-20
4. Consensus meeting: 2026-03-21
5. Top 2 shortlist interviews: 2026-03-22 to 2026-03-25
6. Final decision: 2026-03-26
7. Contract award: 2026-03-27
**Decision Criteria:**
- Minimum score: 70/100 to be considered
- Top 2 scoring vendors invited for interview (technical deep-dive)
- Final decision based on: score (70%) + interview (20%) + cost (10%)
- Tie-breaker: Company with strongest PCI-DSS experience
## Audit Trail Requirements
- All scores documented with justification
- Evaluation notes for each vendor
- Consensus meeting minutes
- Interview scoring sheets
- Final decision rationale
Mode 2: Score Individual Vendor
Scoring Process
- Vendor directory created:
projects/{project}/vendors/{vendor-name}/
- Proposal documents read (if available in directory)
- Interactive scoring:
- Ask for proposal highlights
- Ask for concerns/gaps
- Score each category against framework
- Detailed justification for each score
- Requirement traceability (link to BR/FR/NFR IDs)
Example Scoring Output
# Vendor Evaluation: Acme Payment Solutions
**Project**: Payment Gateway Modernization (001)
**Vendor**: Acme Payment Solutions
**Proposal Date**: 2026-03-14
**Evaluated By**: Lead Architect
**Evaluation Date**: 2026-03-18
## Executive Summary
**Overall Score**: 76/100
**Rank**: 2nd of 3 vendors
**Recommendation**: CONSIDER (shortlist for interview)
**Strengths:**
- Strong PCI-DSS expertise (5+ Level 1 projects)
- Good reference projects (3 UK banks)
- Competitive pricing (£420k vs £480k average)
**Weaknesses:**
- Limited AWS cloud-native experience (mostly on-prem)
- Aggressive timeline (20 weeks vs 24 week recommendation)
- Team availability concerns (2.5 FTE vs 4 FTE requested)
**Risks:**
- 🟠 **Medium Risk**: Cloud-native architecture inexperience may impact NFR-P-001 (99.99% uptime)
- 🔵 **Low Risk**: Timeline pressure may affect code quality
## Mandatory Qualifications
| Qualification | Status | Evidence |
|---------------|--------|----------|
| PCI-DSS Certification | ✅ PASS | QSA certification #12345, expires 2027-06-30 |
| Financial Services Experience | ✅ PASS | 8 years, 5+ payment projects |
| UK References | ✅ PASS | 3 references provided (Barclays, HSBC, Lloyds) |
| Cloud Certification | ⚠️ CONDITIONAL | AWS Solutions Architect (Associate) - not Professional |
| Team Availability | ❌ FAIL | 2.5 FTE offered vs 4 FTE required |
| Proposal Completeness | ✅ PASS | All sections complete |
**Qualification Result**: ❌ **CONDITIONAL PASS** (2 issues: cloud cert level, team size)
**Action Required**:
- Clarify if team can be scaled to 4 FTE
- Request evidence of AWS cloud-native project delivery
---
## Detailed Scoring
### 1. Technical Approach (28/35 points)
#### Solution Design (12/15 points)
- **Architecture Quality** (4/5): Solid microservices design, good PCI segmentation
- ✅ Meets PRIN-005 (API-first design)
- ✅ PCI network segmentation follows best practices
- ⚠️ AWS architecture lacks cloud-native patterns (ECS proposed, no Lambda/Fargate consideration)
- **Scalability** (4/5): Horizontal scaling via ECS auto-scaling
- ✅ Meets NFR-SC-001 (auto-scaling to 1000 TPS)
- ⚠️ Caching strategy limited (Redis only, no CloudFront CDN)
- **Innovation** (4/5): Standard approach, no standout innovations
- ✅ ML fraud detection (FR-007)
- ❌ No mention of chaos engineering (NFR-R-003)
#### Technical Compliance (8/10 points)
- **MUST Requirements Coverage** (6/7): 14/15 MUST requirements addressed
- ✅ BR-001 (Faster Payments): Covered with API integration
- ✅ BR-002 (PCI-DSS): Strong compliance approach
- ✅ BR-003 (Open Banking): OBIE v3.1 compliant
- ❌ NFR-P-001 (99.99% uptime): Only commits to 99.95% (gap!)
- **NFR Approach** (2/3): Good security, weak performance evidence
- ✅ NFR-S-001 to NFR-S-008 (Security): Excellent
- ⚠️ NFR-P-002 (< 3s processing): No load test evidence provided
#### Technology Choices (8/10 points)
- **Stack Appropriateness** (5/5): Go, PostgreSQL, Redis - solid choices
- ✅ Aligns with PRIN-012 (open source preference)
- ✅ Go performance for high-throughput (NFR-P-003)
- **Justification** (3/5): Limited justification for AWS service choices
- ✅ PostgreSQL vs DynamoDB justified (ACID for payments)
- ⚠️ ECS vs Lambda/Fargate not justified
- ❌ No mention of AWS Well-Architected Framework
---
### 2. Project Approach (16/20 points)
#### Methodology (6/8 points)
- Agile with 2-week sprints
- Good sprint planning approach
- ⚠️ Limited stakeholder engagement (weekly demo vs daily collaboration)
#### Risk Management (5/6 points)
- 12 risks identified (good coverage)
- Mitigation strategies documented
- ⚠️ No mention of PCI QSA early engagement risk
#### Quality Assurance (5/6 points)
- >80% test coverage target
- ✅ PCI penetration testing included
- ⚠️ No chaos engineering / disaster recovery testing
---
### 3. Team Qualifications (18/25 points)
#### Relevant Experience (8/12 points)
- **Domain Expertise** (4/6): Strong PCI-DSS, weak cloud-native
- ✅ 5+ PCI Level 1 projects
- ⚠️ 1 AWS project (mostly on-prem experience)
- **Technology Stack** (4/6): Go, PostgreSQL strong; AWS weak
#### Team Composition (7/8 points)
- **Roles** (4/4): Architect, 2 senior devs, 1 junior, QA, DevOps
- **Availability** (3/4): 2.5 FTE vs 4 FTE requested (gap!)
#### Certifications (3/5 points)
- **PCI-DSS QSA**: ✅ Yes
- **AWS Certified**: ⚠️ Associate (not Professional)
- **Security Clearance**: N/A (not required)
---
### 4. Company Experience (8/10 points)
#### Similar Projects (4/5 points)
- 3 strong UK bank references
- Payment gateway modernization experience
- ⚠️ References are on-prem, not cloud
#### Industry Expertise (3/3 points)
- 8 years in payment systems
- Good reputation in UK FinTech
- ISO 9001 quality certification
#### Financial Stability (1/2 points)
- 8-year-old company (✅ stable)
- ⚠️ No financials provided (requested)
---
### 5. Pricing (6/10 points)
#### Cost Competitiveness (3/5 points)
- **Total Cost**: £420k (vs £480k average, £540k high)
- **Benchmark**: 12% below average (good value)
- ⚠️ Lowest bid (may indicate underestimation)
#### Pricing Clarity (2/3 points)
- Transparent breakdown by phase
- ⚠️ "Cloud infrastructure costs" TBD (hidden cost risk)
#### Commercial Terms (1/2 points)
- Milestone payments aligned
- ⚠️ Only 30-day warranty (vs 90-day requested)
---
## Risk Assessment
| Risk Area | Risk Level | Mitigation |
|-----------|------------|------------|
| **Cloud-native delivery** | 🟠 MEDIUM | Request AWS reference project proof |
| **Team capacity** | 🟠 MEDIUM | Clarify if team can scale to 4 FTE |
| **Uptime SLA gap** | 🟠 MEDIUM | Negotiate 99.99% or accept 99.95% |
| **Hidden cloud costs** | 🔵 LOW | Request fixed-price cloud infrastructure |
| **Aggressive timeline** | 🔵 LOW | Add 2-week contingency buffer |
**Overall Risk**: 🟠 **MEDIUM** (proceed with caution, clarify gaps)
---
## Recommendation
**CONSIDER** - Shortlist for technical interview
**Rationale**:
- ✅ Strong PCI-DSS expertise (critical for BR-002)
- ✅ Competitive pricing (12% below average)
- ✅ Good UK FinTech references
- ⚠️ Cloud-native experience gap needs validation
- ⚠️ Team capacity issue (2.5 vs 4 FTE)
- ⚠️ Uptime SLA gap (99.95% vs 99.99%)
**Interview Focus Areas**:
1. AWS cloud-native architecture deep-dive (request examples)
2. Team scaling plan to 4 FTE
3. Uptime SLA negotiation (99.99% commitment)
4. Load testing approach and evidence
5. Cloud cost estimation methodology
**Contract Negotiation Points** (if selected):
- Increase warranty to 90 days
- Fixed-price cloud infrastructure (no TBD costs)
- Commit to 99.99% uptime SLA with financial backing
- Minimum 4 FTE availability
- AWS Well-Architected review at HLD stage
Mode 3: Compare Vendors
Comparison Matrix
Side-by-side comparison table:
# Vendor Comparison: Payment Gateway Modernization
**Project**: 001 - Payment Gateway Modernization
**Document**: ARC-001-VEND-v1.0
**Vendors Evaluated**: 3
**Date**: 2026-03-20
## Executive Summary
**Recommendation**: **BestPay Solutions** (82/100)
**Shortlist for Interview**: BestPay Solutions (82), Acme Payment Solutions (76)
**Eliminated**: CloudPayments Inc (71) - below 75-point threshold
---
## Overall Scores
| Rank | Vendor | Total Score | Technical | Project | Team | Experience | Pricing | Recommendation |
|------|--------|-------------|-----------|---------|------|------------|---------|----------------|
| 🥇 1st | **BestPay Solutions** | **82/100** | 33/35 | 18/20 | 22/25 | 9/10 | 0/10 | ✅ **RECOMMEND** |
| 🥈 2nd | Acme Payment Solutions | 76/100 | 28/35 | 16/20 | 18/25 | 8/10 | 6/10 | ⚠️ **CONSIDER** |
| 🥉 3rd | CloudPayments Inc | 71/100 | 24/35 | 14/20 | 20/25 | 7/10 | 6/10 | ❌ **NOT RECOMMENDED** |
---
## Detailed Comparison
### Technical Approach (35 points max)
| Criterion | BestPay | Acme | CloudPayments |
|-----------|---------|------|---------------|
| Solution Design (15) | 14 | 12 | 10 |
| Technical Compliance (10) | 10 | 8 | 7 |
| Technology Choices (10) | 9 | 8 | 7 |
| **Subtotal** | **33** | **28** | **24** |
**Winner**: **BestPay Solutions**
- Best cloud-native architecture (Lambda, Fargate, serverless)
- 15/15 MUST requirements covered (Acme: 14/15, Cloud: 13/15)
- AWS Well-Architected Framework applied
**Acme Strengths**:
- Strong PCI segmentation
- Good microservices design
**Acme Weaknesses**:
- NFR-P-001 gap (99.95% vs 99.99% uptime)
- ECS-only (not cloud-native)
**CloudPayments Weaknesses**:
- Monolithic architecture (not microservices)
- 2 MUST requirements not addressed
---
### Team Qualifications (25 points max)
| Criterion | BestPay | Acme | CloudPayments |
|-----------|---------|------|---------------|
| Relevant Experience (12) | 10 | 8 | 11 |
| Team Composition (8) | 8 | 7 | 6 |
| Certifications (5) | 4 | 3 | 3 |
| **Subtotal** | **22** | **18** | **20** |
**Winner**: **BestPay Solutions**
- AWS Certified Solutions Architect (Professional) ✅
- 4 FTE committed ✅
- 3 AWS cloud-native payment projects
**Acme**: Strong PCI-DSS, weak cloud (Associate cert, 2.5 FTE)
**CloudPayments**: Good experience, small team (3 FTE, availability concerns)
---
### Pricing (10 points max)
| Vendor | Total Cost | Points | Notes |
|--------|------------|--------|-------|
| Acme Payment Solutions | £420,000 | 6/10 | Lowest bid (12% below avg), TBD cloud costs |
| CloudPayments Inc | £480,000 | 6/10 | Average price, transparent |
| **BestPay Solutions** | £540,000 | 0/10 | **Highest bid (12% above avg)**, but best value |
**BestPay Pricing Justification**:
- 28% more than Acme (£540k vs £420k)
- BUT: Includes fixed-price cloud infrastructure (Acme = TBD)
- 99.99% uptime SLA (Acme = 99.95%)
- 4 FTE (Acme = 2.5 FTE)
- 90-day warranty (Acme = 30 days)
- **Adjusted TCO**: BestPay £540k vs Acme £480k (with cloud costs + team scaling)
---
## Strengths Comparison
### BestPay Solutions
- ✅ **Best technical approach**: Cloud-native, serverless, AWS Well-Architected
- ✅ **Best cloud experience**: 3 AWS payment projects, Professional cert
- ✅ **Best compliance**: Meets all 15 MUST requirements + 99.99% SLA
- ✅ **Best team**: 4 FTE, AWS expertise, good availability
- ✅ **Best commercial terms**: 90-day warranty, fixed cloud costs
- ❌ **Highest price**: £540k (but best value when adjusted for TCO)
### Acme Payment Solutions
- ✅ **Best PCI-DSS expertise**: QSA cert, 5+ Level 1 projects
- ✅ **Best pricing**: £420k (lowest bid)
- ✅ **Good UK references**: 3 UK banks (Barclays, HSBC, Lloyds)
- ⚠️ **Weak cloud-native**: ECS-only, Associate cert, 1 AWS project
- ⚠️ **Team capacity gap**: 2.5 FTE vs 4 FTE required
- ⚠️ **SLA gap**: 99.95% vs 99.99% uptime
### CloudPayments Inc
- ✅ **Good domain experience**: 6 payment projects
- ✅ **Transparent pricing**: £480k, clear breakdown
- ❌ **Weak architecture**: Monolithic, not microservices
- ❌ **Below threshold**: 71/100 (< 75 required)
- ❌ **2 MUST requirements**: Not addressed (BR-003, NFR-P-001)
---
## Risk Comparison
| Risk Area | BestPay | Acme | CloudPayments |
|-----------|---------|------|---------------|
| **Technical Delivery** | 🟢 LOW | 🟠 MEDIUM | 🔴 HIGH |
| **Cloud-Native** | 🟢 LOW | 🟠 MEDIUM | 🔴 HIGH |
| **PCI-DSS Compliance** | 🟢 LOW | 🟢 LOW | 🟠 MEDIUM |
| **Team Capacity** | 🟢 LOW | 🟠 MEDIUM | 🟠 MEDIUM |
| **Uptime SLA** | 🟢 LOW | 🟠 MEDIUM | 🔴 HIGH |
| **Cost Overrun** | 🟢 LOW | 🟠 MEDIUM | 🟢 LOW |
| **Overall Risk** | 🟢 **LOW** | 🟠 **MEDIUM** | 🔴 **HIGH** |
---
## Final Recommendation
### 🥇 Selected Vendor: **BestPay Solutions**
**Score**: 82/100 (highest)
**Decision Rationale**:
1. **Best Technical Fit**: Only vendor meeting all 15 MUST requirements with 99.99% SLA
2. **Lowest Technical Risk**: Proven AWS cloud-native expertise (3 payment projects)
3. **Best Long-Term Value**: £540k includes fixed cloud costs, 90-day warranty, 4 FTE
4. **Principle Alignment**: Strongest alignment with PRIN-001 (cloud-first) and PRIN-005 (API-first)
5. **Quality Over Cost**: 28% price premium justified by risk reduction and compliance
**Contract Negotiation**:
- Accept £540k pricing (justified by TCO analysis)
- Confirm 99.99% SLA with financial backing (SLA credits)
- Lock in 4 FTE availability (contractual commitment)
- Include AWS Well-Architected review at HLD stage
### 🥈 Runner-Up: Acme Payment Solutions
**Fallback Option If**: BestPay negotiations fail or budget constraints
**Required Clarifications**:
- Scale team to 4 FTE (increase £420k → £480k estimate)
- Commit to 99.99% uptime SLA
- Provide AWS cloud-native project evidence
- Fix cloud infrastructure costs (no TBD)
**Adjusted Score** (if gaps addressed): 81/100 (competitive)
### ❌ Not Recommended: CloudPayments Inc
**Reasons**:
- Below 75-point threshold (71/100)
- 2 MUST requirements not addressed (disqualifying gaps)
- High technical risk (monolithic architecture)
---
## Audit Trail
**Evaluation Panel**:
- Lead Architect (scoring + recommendation)
- Senior Engineer (technical review)
- Product Owner (business fit)
- Procurement Manager (commercial review)
**Consensus Meeting**: 2026-03-21, 14:00-16:00 GMT
**Attendees**: [List]
**Decision**: Unanimous recommendation for BestPay Solutions
**Dissenting Views**: None
**Interview Outcome** (if conducted):
- BestPay: Excellent technical deep-dive, AWS architecture validated
- Acme: Good PCI-DSS discussion, cloud-native concerns confirmed
**Final Approval**: [Pending] - Architecture Board, 2026-03-26
Evaluation Best Practices
Objectivity
- ✅ Documented criteria before receiving proposals
- ✅ Specific justification for each score (no arbitrary numbers)
- ✅ Requirement traceability (link to BR/FR/NFR IDs)
- ✅ Multiple evaluators (reduce bias)
- ✅ Audit trail (meeting minutes, scoring sheets)
Mandatory Qualifications
- ✅ Pass/fail (missing any = disqualified)
- ✅ Checked first (before detailed scoring)
- ✅ Evidence required (certificates, references)
- ✅ No exceptions (maintains fairness)
Scoring Transparency
- ✅ Justification required for each score
- ✅ Reference requirement IDs (BR-001, NFR-P-001)
- ✅ Strengths and weaknesses documented
- ✅ Risk assessment per vendor
- ✅ Interview notes if conducted
UK Public Sector Specific
Social Value (10% weighting):
- Apprenticeships and skills development
- SME subcontracting commitments
- Environmental sustainability
- Local economic impact
Evaluation Panel:
- Technical authority (RACI: Accountable)
- Commercial lead (RACI: Accountable)
- User representative (RACI: Consulted)
- Finance (RACI: Informed)
Audit Requirements:
- Evaluation report for approving authority
- Decision rationale documented
- Conflicts of interest declared
- Award notice published (Contracts Finder)