# I Tested AI on GitLab's Public Security Questionnaire

Source: https://cplt.tech/blog/ai-security-questionnaire-caiq-gitlab-test
Author: Stéphane Lepain
Published: 2026-10-13T09:00:00Z
Updated: 2026-10-15T14:23:41Z
Tags: security-questionnaire, caiq, compliance, ai-governance

---

Updated 15 October 2026: corrected the review step and its figures.

**Short answer: yes, mostly, with real limits.** On 236 unseen questions, an AI workflow matched GitLab's own published answers 74% of the time working from GitLab's public documents alone, said "not enough evidence" instead of guessing on another 25%, and sent 27% of its answers to an expert for review. It never confirmed a single "No," and both of its wrong answers went into the ready-to-sign-off pile, because each carried a genuine quote.

Companies that sell software answer the same security questionnaire again and again, once per customer. The standard one from the Cloud Security Alliance, the CAIQ, has 283 questions. I wanted to know whether an automated workflow could draft real answers to it from nothing but a company's own public documents, and whether I could trust the result enough to hand it to a reviewer.

**I took the CAIQ that GitLab completed and published for GitLab.com, hid GitLab's answers, and had a workflow answer every question using only GitLab's public security handbook.** Each answer had to quote its source and link to it. A short program, not an AI model, then checked that every quote appears word for word in the page it cites, before I compared anything with what GitLab actually said.

On the 236 questions the workflow had never seen before, it answered 176 and matched GitLab's own answer on 174 of them. On the other 60 it said "not enough evidence in the public documents" instead of guessing. The whole run took 15 minutes of machine time.

## What I'm actually testing

The questionnaire: CSA CAIQ v4.1.0, 283 Yes/No/N/A questions across 17 domains. The answer key: GitLab.com's own completed CAIQ, published on the [CSA STAR registry](https://cloudsecurityalliance.org/star/registry/gitlab/services/gitlab) and renewed 2026-10-08 (260 Yes, 8 No, 15 N/A). The evidence: [GitLab's public security handbook](https://handbook.gitlab.com/handbook/security/), 426 source files, frozen at a single commit dated 2026-10-12. Everything here is public data, attributed to GitLab and the Cloud Security Alliance. Nothing in this article says anything about GitLab's actual security posture: it tests whether a public handbook can answer a public questionnaire, not whether GitLab is secure.

I checked first that the handbook doesn't already contain GitLab's CAIQ answers. It mentions the CAIQ only in passing, in its own supplier-review policy. The workflow never saw GitLab's answers or explanations until scoring.

## How it works

1. The handbook's substantive pages are split into passages, then searched by keyword and by meaning for each question.
2. An AI model drafts the answer, Yes, No, N/A, or "not enough evidence", from those passages only, with a short justification, the exact quote and the source link.
3. If the evidence looks thin, the model writes one follow-up search and tries again, the way an analyst would.
4. Plain code checks that the quote appears word for word in the cited passage.
5. An answer goes to the expert review queue when it says "not enough evidence" or its quote is not found word for word. Everything else goes to a "ready to sign off" pile.
6. Only then did I compare the result against GitLab's published answers, which stayed hidden until this step.

## Keeping the test honest

I tried this first on a 47-question set covering identity and access, and data-centre security, the hardest part of the questionnaire. It was too cautious: 25 of 47 came back "not enough evidence", and only 16 of 47 matched. The cause was my search, not the handbook: for multi-factor authentication, for example, the evidence was there but the first search missed it. I widened the search and added the one follow-up step. On the same 47 questions, abstentions dropped to 11 and matches rose to 27.

**Because I tuned the method on those 47 questions, they don't count in the headline.** The numbers below come only from the other 236 questions, run once, with the method frozen before the run. I also fixed the scoring rules before the run: exact label match against GitLab, the "always answer Yes" baseline reported alongside, and every disagreement listed.

**One thing changed after the run: the rule that decides which answers go to the expert.** My first routing rule relied on an automated rating of the evidence. Measured on this data, it was no better than picking answers at random, so I replaced it with the quote check above and re-scored the same 236 answers. The answers did not change. The routing figures did. Because I made that decision after seeing the held-out results, the routing figures below (sign-off, review queue, wrong answers sent to review) are a re-scoring, not a frozen-method result. The answer figures (176 answered, 174 matched, 60 "not enough evidence") are not affected.

## Results on the 236 questions

The routing rows in this table (ready to sign off, sent to the expert, wrong answers sent to review) are the re-scoring described above.

![236 questions, answered from public documents: 174 of 236 matched GitLab, 2 of 236 differed, 60 of 236 came back not enough evidence](https://cplt.tech/images/blog/caiq-case-study/01-outcome.webp)

| Measure | Result |
| --- | --- |
| Answered (Yes/No/N/A) | 176 of 236 (75%) |
| "Not enough evidence", not guessed | 60 of 236 (25%) |
| Answers matching GitLab | 174 of 176 (99%) |
| Matching GitLab, counting abstentions as misses | 174 of 236 (74%) |
| Baseline "answer Yes to everything" | 225 of 236 (95%), labels only, no evidence |
| No / N/A cases answered correctly | 0 of 11 |
| Wrong answers | 2 of 176 |
| Quote found word for word in the source | 173 of 176 (98%) |
| Ready to sign off | 173 of 236 (73%); 171 of 173 matched GitLab |
| Sent to the expert for review | 63 of 236 (27%): 60 "not enough evidence", 3 quotes not found |
| Wrong answers sent to review | 0 of 2 |
| Machine time | 917 s = 15.3 minutes; 3.9 s per question |

![Every answer shows its evidence: 176 of 176 answers cite a source page, 173 of 176 quotes found word for word, 3 of 176 quotes not found and sent to review](https://cplt.tech/images/blog/caiq-case-study/03-evidence.webp)

![Where the expert's time goes: 173 of 236 ready to sign off, 63 of 236 sent to the expert for review, 0 of 2 wrong answers sent to review](https://cplt.tech/images/blog/caiq-case-study/02-review-queue.webp)

![Time to a first draft: 15 minutes for 236 questions, 3.9 seconds per question, 63 left for the expert to review](https://cplt.tech/images/blog/caiq-case-study/04-time.webp)

## What I will not claim

This result is better than "just answer Yes", but not on the scorecard alone. **Answering Yes to every question on this questionnaire scores 225 of 236 (95%) on labels, because GitLab's real answers are 95% Yes.** My workflow matched 174 of 236 (74%) counting its abstentions as misses. The reason to prefer it isn't the label count: it's that every answer carries a quote and a link, and the gaps are flagged instead of invented.

**It does not confirm a "No."** Of the 11 held-out questions where GitLab answered No or N/A, the workflow got zero right: it said "not enough evidence" on 9 of them, which sent them safely to a human, and on 2 it answered Yes when GitLab's own answer was No or N/A. Public documents describe what a company does. Proving what it doesn't do usually needs evidence the public doesn't see, the same limit I ran into when [checking a supplier's named versus assumed security controls](https://cplt.tech/blog/named-or-assumed-architecture-review): a document that stays silent on a control isn't proof the control is absent.

**Both wrong answers went into the ready-to-sign-off pile.** On UEM-14.1 (security of third-party endpoints), GitLab answered N/A ("no third-party endpoints have access"). The workflow answered Yes from a handbook passage that does require endpoint controls for third parties. On UEM-02.1, GitLab answered No and the workflow answered Yes. Both answers carry a genuine quote from the cited page, so the quote check passes them. The check proves the evidence exists. It does not prove the conclusion is right, and it is not a correctness check.

**It does not replace the expert, who still signs off every answer.** 63 of 236 answers (27%) go to the review queue first: 60 where the evidence was not enough and 3 where the quote could not be found word for word. The other 173 are ready for sign-off, and 171 of them matched GitLab. The two that did not are the wrong answers above, which is why the expert reads the ready pile too, not only the queue. A real deployment would tune the review rule against an expert's actual corrections, which this demo didn't do.

**The 47-question tuning set is not part of this headline.** It's the hardest part of the questionnaire (12 of 47 are No/N/A, mostly physical data-centre controls) and I used it to fix my own method before freezing it, so I report it separately, not folded into the 236.

## Two real examples

**CEK-10.1** asks whether cryptographic keys are generated using approved cryptographic libraries. GitLab's own answer: Yes. The workflow answered Yes too, with the quote found word for word, citing GitLab's Cryptographic Standard, which names the approved modules and sets minimum key sizes.

**STA-09.1** asks whether a Bill of Materials process is enforced for the service supply chain. GitLab's own answer: No. The workflow searched again, found that the handbook lists a company-wide SBOM as a goal still to be delivered rather than a process already enforced, and answered "not enough evidence" instead of guessing Yes. That's the honest abstention working as intended: a gap flagged, not invented.

## What this means if you fill these out yourself

A first draft that cites its source for every answer, flags what it can't support, and has every quote checked against its page before a person ever opens it, changes where an expert's hour goes: fewer questions read cold, more time on the ones the evidence doesn't settle. That's the same discipline I want from any AI assistant I build: [show the evidence, don't just trust the output](https://cplt.tech/proof). That's the premise I'd want to test against your own questionnaire and your own documents, not just GitLab's.

If your team answers security questionnaires from your own policies and evidence and you want to see whether this holds up on your documents, [get in touch](https://cplt.tech/contact) and we can scope a short test the same way I scope any [AI assistant or automation build](https://cplt.tech/services): fixed written scope, agreed human checks, no promised accuracy rate.

## Questions people ask about AI and security questionnaires

**Can AI fill out a CAIQ or security questionnaire accurately?**
On this test, an AI workflow matched GitLab's own published answers on 174 of 236 unseen questions (74%, counting its abstentions as misses). It sent 27% of its answers to an expert for review, the expert still signs off every answer, and it could not confirm a single "No" or "N/A."

**Does this replace the person who reviews security questionnaires?**
No. 63 of 236 answers went to the expert review queue first, and both wrong answers passed the quote check, because each carried a genuine quote. The expert signs off every answer, not only the ones in the queue.

**What happens when the public evidence doesn't cover a question?**
The workflow is built to say "not enough evidence" instead of guessing. That happened on 60 of the 236 questions (25%).

**Does this only work for GitLab, or for any company's questionnaire?**
The method is general: feed it a questionnaire and a set of documents, public or internal. This test used GitLab's published CAIQ because GitLab publishes both the questionnaire and the real answers, so the result could be checked against the truth.

**Is the CSA CAIQ the only questionnaire this applies to?**
No. CAIQ is this test's example because its answer key is public. The same method applies to a SIG, a VSA, or a customer's own security questionnaire, as long as there's a document set to answer it from.