10x-Modern-Contract-Vehicles

18F project to understand how our team can better support colleagues, partners, and vendors in the procurement of agile development and human-centered design services.

Why we’re here

This is a Phase Two continuation of Modern Contract Vehicles, with the goal of identifying how TTS can better support colleagues, partners, and vendors in the procurement of agile development and human-centered design services.

The Agile BPA began as three different vendor pools, one for full stack development that was ultimately awarded; one for development skills as a small business set aside and one for design, both of which did not make it through to award. With the pool that was ultimately awarded, there were a few core issues: time to award, competition, user research resources, and onboarding and offboarding vendors. Out of our 10 orders in two years, the average amount of bids was 6. We saw a steady decline in competition as well. The first few task orders (FedRAMP marketplace, eQIP, 14(c)) all received high amount of bids, 7-14. Our most recent task orders (Christmas tree permitting and the second 14(c) project) saw only 2 and 5 bids respectively. Without adequate competition amongst the vendors on the BPA, 18F was no longer confident that we were getting the strongest vendors for our agency partners. This is detailed publicly in our blog post.

Phase One focused internally on TTS Acquisition and evaluated whether the Agile BPA was helping TTS achieve our business goals and improve outcomes for our partner agencies. The research showed that the Agile BPA does not necessarily improve our ability to evaluate and contract high quality professional services in the areas of agile software development and human-centered design.

While the findings were inconclusive about the value of vendor pools, it surfaced two broader/adjacent questions requiring further investigation:

How do we identify high-quality vendors for agile development and HCD and ensure that they respond to our solicitations?
What requirements, criteria, and evaluation methods are most effective in selecting vendors who can deliver working software on time, on budget, and in alignment with agile, open, and user-centered practices?

These questions are the focus of Phase Two. The project team will broaden its research to include external agency partners, acquisition experts, and vendors.

The project team will document and share learnings and results from various vendor evaluation and modular contracting mechanisms that have been tried across government (vendor pools, public Github repos, verbal interviews, etc). Findings and recommendations may be further tested with prototypes to improve how federal agencies hire private vendors to build products and services using agile development techniques.

Research Questions

What requirements, criteria, and evaluation methods are most effective in selecting vendors who can deliver working software on time, on budget, and in alignment with agile, open, and user-centered practices?

Are expected capabilities/skills for these disciplines clearly defined? Are partners familiar with them, and/or existing resources?
What vendor evaluation methods have been successful for identifying contractors who have strong human-centered design capabilities?
What vendor evaluation methods have been successful for identifying contractors who can successfully use open practices?
What vendor evaluation methods do stakeholders feel frustrated by? Or seem ineffective for evaluating specific skills?
How confident do partners feel in their ability to evaluate vendors through various methods? What would help them do this better?

How do we identify high-quality vendors for agile development and HCD and ensure that they respond to our solicitations?

Can vendor pools help deliver software in an agile way across government?
What vendor pools currently exist for agile delivery services? Are they operating similarly? Any standouts that warrant deeper exploration?
Are there any vendor pools currently focused on a targeted/niche discipline? Which disciplines?
Do experiences with targeted/niche pools vary by discipline?
Which specific niche skill sets, common to many of our partners’ projects, are most difficult to source? Why? Which skills are easiest to source?
Does contracting off of vendor pools have an impact on project timelines? Administrative burden?
How do outcomes for projects contracted off of pools compare to open solicitations? Does a targeted/niche vendor pool improve project outcomes?
How often is market research done prior to solicitation? How is this research done among project teams and within contracting shops?
What other methods / mechanisms are COs and Project Teams using to identify potential vendors and get quality responses to solicitations? Which have returned higher numbers of quality responses? What are they doing differently than traditional contracting process?

Research period

Desk research: Week 1-2
Stakeholder interviews: Week 2-5
Synthesis: Week 5-6

Research Methods

Desk review of existing resources for agile/HCD procurements
Desk review of existing contract vehicles
Stakeholder Interviews
- 18F
- Partner Agencies (Contracting Officers & Program Staff)
- Vendors
Journey mapping with agency partners
Journey mapping with vendors

Interview Questions

18F

What roles/disciplines are essential to agile/open/HCD work (in your projects/agency)?
What specific capabilities/skills are you looking for (or should agency partners be looking for) from vendors for agile development & HCD?
How much do the desired skills/capabilities vary from project to project?
How have you assessed/evaluated for these skills in potential vendors? What methods have you found to be most successful? Which do you use most frequently and why?
Ask follow-ups about specific methods such as research challenge, presentation-style interview, etc.
What are some red flags?
Are there any methods, criteria, or interview questions you use regularly?
Which skills do you think are most difficult to evaluate for?
Where do you think our current evaluation methods succeed/fall short?
In your opinion, do agency partners have confidence in evaluating vendors in HCD and agile development skills?
How have you advised/supported partners during the vendor evaluation process? What needs/concerns/challenges have come up?
Once a vendor has been contracted, is there a point in project delivery when you can usually tell whether it’s the right fit and whether that vendor is able to perform up to 18F standards? How can you tell?
Are there common areas of weakness you’ve seen in Agile BPA vendors or others related to HCD, Agile dev, and working open?
Who else should we talk to?

Partners

Can you talk us through a recent IT contracting process - SOW, timeframe, eval methods, challenges/frustrations, and outcomes/current status?
Did you get what you expected from (x contract)? (Was project on time? on budget? when did you first get working software? was user research and usability testing done?)
What roles/disciplines are essential to agile/open/HCD work (in your projects/agency)?
What specific capabilities/skills are you looking for from vendors for agile dev & HCD?
How have you assessed/evaluated for these skills in potential vendors? What methods have you found to be most successful? Which do you use most frequently? Why?
Ask follow-ups about research challenge, presentation-style interview, technical prototype
What are some red flags that a vendor may not be able to work in an agile way? Or may not be considering users & user testing?
In your experience, which skills are most difficult to evaluate? Where do you think current evaluation methods succeed/fall short?
In your experience, what skills are most difficult to source for? Why?
How confident do you feel in evaluating skills/competencies in area of: Agile development, HCD, working open? What do you look for?
Have you seen any correlation between how vendors are evaluated and how they perform on projects? Can you give a specific example?
Once a vendor has been contracted, is there a point in project delivery when you can usually tell whether it’s the right fit and whether that vendor is able to deliver value frequently and consistently (in a way that a traditionally contracted vendor would not)? How can you tell?
What options do you have if it is not? Have you ever had a situation where you had to change key personnel or terminate the vendor?
To what extent have you worked with vendor pools? Which one(s) have you used? Do you know of any targeted/niche vendor pools that your agency uses?
Has the contracting process and admin burden differed between small vendor pools, larger pools, and open competition?
How do you usually do pricing for agile dev contracts? (i.e. Time and Materials, fixed price)
At your agency, are IT funds earmarked for either “Development” or “Operations & Maintenance” acquisitions?
To what extent have module contracting / agile development / HCD permeated practices at your shop?
- What has been turning point for uptake of these methods?
- (or) what is inhibiting people from using them? What do you think would help?

COs

Can you talk us through a recent IT contracting process - SOW, timeframe, eval methods, challenges/frustrations, and outcomes/current status?
What capabilities and skills do you look for (or should government look for) when soliciting vendors for agile software development? For human-centered design?
What do you do differently throughout the acquisition process to support agile practices instead of waterfall?
Can you talk a little bit about how you frame evaluation criteria and the project’s/ requirements in the RFP, i.e., as technical requirements or as outcomes?
Among the government project teams and product owners you’ve worked with, what is the level of familiarity with agile development and human centered design?
How do you involve the program team in the acquisition process?
Working on IT acquisitions, where do you look for information, and which resources have you found to be helpful?
What resources do you have that don’t work as well as you want them to? How could they be better?
What resources do you wish you had access to? What do you wish you knew?
To what extent have you worked with vendor pools? Which one(s) have you used? Do you know of any niche vendor pools that other agencies use?
Are there any documents/deliverables that you can point us to about this specific project??
(For DITAP alum): What was your experience with DITAP, and has it changed aspects of your acquisition process? If so, what has changed?
Who else we should talk to?

Vendors

Can you walk us through your experience with the a recent bid process? (sketch-up journey-map during discussion)
- How did you find out about the solicitation?
- Why did you respond? What factors did you consider when deciding whether or not to respond?
- How did you go about the proposal-writing process?
- Did you know what to expect for the technical evaluation and interview (if there was one)? What did you expect, and what did you think the evaluation panel was looking for?
- Which capabilities did you focus on? How did you demonstrate them?
- Did you get feedback from the evaluation panel? What was their feedback?
- Do you feel you had a chance to accurately demonstrate/portray your capabilities/skills?
What do you think is the most effective way for your team to demonstrate your capabilities? Are there methods you’ve seen within government or elsewhere that you particularly liked? Why?
Do you think evaluation panels have understood your work? Have they seemed knowledgeable about the skills/competencies you are demonstrating?
Which skills have been requested most often on government solicitations you have responded to?
Which skills have you ended up using most often (or have been most critical) for government contracts you have been awarded?
How do you determine key personnel?
Have there been instances where you were unable to share previous work samples b/c of privacy/security concerns with a partner? How else have you demonstrated your team’s skills (or what do you think would work)?
What challenges or frustrations have you faced as a vendor? Are there parts of the process that have felt particularly opaque or where something played out much differently than you expected?
How many contracts have you gotten as part of smaller vendor pools vs. open competition or broader vendor pool? Has the contracting process differed between pools of different sizes /open competition?

bustosfj / 10x-modern-contract-vehicles Goto Github PK