Skip to content

Commit

Permalink
Adding new survey question and improving logic for capturing answers
Browse files Browse the repository at this point in the history
  • Loading branch information
@mageroni
mageroni committed Jul 26, 2024
1 parent 5e59bc4 commit d8bdd82
Show file tree
Hide file tree
Showing 12 changed files with 1,296 additions and 1,267 deletions.
11 changes: 9 additions & 2 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@

As more companies adopt GitHub Copilot, it becomes increasingly important to measure the benefits it brings to the organization. This survey is an effort to combine both quantitative and qualitative data. To improve validity of the quantitative responses, Developers are asked to document their rationale for the time-savings percentage they choose.

Quantitative feedback from the Developer at the time of creating a PR provides valuable insights on the time savings experienced by the Developer. Time savings is needed first before other downstream impacts (like velocity increases, or other improvements can happen. The level of granularity provides multiple feedback opportunities for Developers and can capture a variety of PRs so we can understand adoption challenges and improvement opportunities. If helpful, the Survey results may also be combined with Key Performance Indicators (KPIs) that the product provides to further contextualize the survey responses.
Quantitative feedback from the Developer at the time of creating a PR provides valuable insights on the time savings experienced by the Developer. Time savings is needed first before other downstream impacts like velocity increases, or other improvements can happen. The level of granularity provides multiple feedback opportunities for Developers and can capture a variety of PRs so we can understand adoption challenges and improvement opportunities. If helpful, the Survey results may also be combined with Key Performance Indicators (KPIs) that the product provides to further contextualize the survey responses.

The survey responses are stored in your private Azure SQL database to provide insights into how developers are using the tool, the value they report, and the challenges they encounter.
The survey responses are stored in a file called results.csv in a new branch "copilot-survey-engine-results" to provide insights into how developers are using the tool, the value they report, and the challenges they encounter.

We hope that this project provides value to your organization, and we encourage you to contribute and build upon it. Your contributions can help further enhance the survey capabilities and provide even greater insights into the developer experience with Copilot.

Expand Down Expand Up @@ -55,6 +55,13 @@ Note: *If the env file does not contain a Language API Key or Endpoint, the anal

- [ replace this line with your answer. ]

6. ***Where did you invest your Copilot Time Savings?***
- [ ] Resolve vulnerabilites
- [ ] Experiment, Learn and Wellness
- [ ] Technical debt and refactorization
- [ ] Work on other items in the backlog
- [ ] Other. Please explain in the comment

### Where does the app store surveys?

As we receive edits on the issue, the App will validate the responses received (options selected) and once all questions have been answered, the issue will be closed automatically and the responses will be saved into a results.csv file in the same repo in which the issue was created.
Expand Down
227 changes: 104 additions & 123 deletions index.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,10 +65,6 @@ module.exports = (app) => {

async function GetSurveyData(context) {
let issue_body = context.payload.issue.body;
let pctSelected = false;
let pctValue = new Array();
let freqSelected = false;
let freqValue = new Array();

// save comment body if present
let comment = null;
Expand All @@ -79,56 +75,49 @@ module.exports = (app) => {
// find regex [0-9]\+ in issue_body and get first result
let pr_number = issue_body.match(/[0-9]+/)[0];

// find regex \[x\] in issue_body and get complete line in an array
let checkboxes = issue_body.match(/\[x\].*/g);

// find if checkboxes array contains Sim o Si or Yes
let isCopilotUsed = checkboxes.some((checkbox) => {
// Get answers to first question and find if they contain affirmative answer
let firstQuestionResponse = await getQuestionResponse(1, 2, issue_body);
let isCopilotUsed = firstQuestionResponse.some((response) => {
return (
checkbox.includes("Sim") ||
checkbox.includes("Si") ||
checkbox.includes("Yes") ||
checkbox.includes("Oui")
response.includes("Sim") ||
response.includes("Si") ||
response.includes("Yes") ||
response.includes("Oui")
);
});

if (isCopilotUsed) {
// loop through checkboxes and find the one that contains %

for (const checkbox of checkboxes) {
if (checkbox.includes("%")) {
pctSelected = true;
copilotPercentage = checkbox;
copilotPercentage = copilotPercentage.replace(/\[x\] /g, "");
pctValue.push(copilotPercentage);
app.log.info(copilotPercentage);
}
}
// Get answers to second question and store in pctValue
let pctValue = await getQuestionResponse(2, 3, issue_body);
let freqValue = await getQuestionResponse(4, 5, issue_body);
let savingsInvestedValue = await getQuestionResponse(6, '', issue_body);

// loop through checkboxes and find the ones that do not contain % and are not Yes or No
for (const checkbox of checkboxes) {
if (
!checkbox.includes("%") &&
!checkbox.includes("Sim") &&
!checkbox.includes("Si") &&
!checkbox.includes("Yes") &&
!checkbox.includes("Oui") &&
!checkbox.includes("Não") &&
!checkbox.includes("No") &&
!checkbox.includes("Non") ||
checkbox.includes("Not very much")
) {
freqSelected = true;
frequencyValue = checkbox;
frequencyValue = frequencyValue.replace(/\[x\] /g, "");
freqValue.push(frequencyValue);
app.log.info(frequencyValue);
}
if( isCopilotUsed && pctValue && freqValue && savingsInvestedValue){
// All questions have been answered and we can close the issue
app.log.info("Closing the issue");
try {
await context.octokit.issues.update({
owner: context.payload.repository.owner.login,
repo: context.payload.repository.name,
issue_number: context.payload.issue.number,
state: "closed",
});
} catch (err) {
app.log.error(err);
}

if( pctSelected && freqSelected ){
// close the issue
}

if (
firstQuestionResponse.some((response) => {
return (
response.includes("Não") ||
response.includes("No") ||
response.includes("Non")
);
})
){
if (comment) {
try {
// close the issue
await context.octokit.issues.update({
owner: context.payload.repository.owner.login,
repo: context.payload.repository.name,
Expand All @@ -139,33 +128,8 @@ module.exports = (app) => {
app.log.error(err);
}
}
} else {
if (
checkboxes.some((checkbox) => {
return (
checkbox.includes("Não") ||
checkbox.includes("No") ||
checkbox.includes("Non")
);
})
) {

if (comment) {
try {
// close the issue
await context.octokit.issues.update({
owner: context.payload.repository.owner.login,
repo: context.payload.repository.name,
issue_number: context.payload.issue.number,
state: "closed",
});
} catch (err) {
app.log.error(err);
}
}
}
}

let data = {
enterprise_name: context.payload.enterprise ? context.payload.enterprise.name : '',
organization_name: context.payload.organization ? context.payload.organization.login : '',
Expand All @@ -175,8 +139,9 @@ module.exports = (app) => {
PR_number: pr_number || '',
assignee_name: context.payload.issue.assignee ? context.payload.issue.assignee.login : '',
is_copilot_used: isCopilotUsed ? 1 : 0,
saving_percentage: pctValue || '',
usage_frequency: freqValue || '',
saving_percentage: pctValue ? pctValue.join(" || ") : '',
frequency: freqValue ? freqValue.join(" || ") : '',
savings_invested: savingsInvestedValue ? savingsInvestedValue.join(" || ") : '',
comment: comment || '',
created_at: context.payload.issue ? context.payload.issue.created_at : '',
completed_at: context.payload.issue ? context.payload.issue.updated_at : ''
Expand All @@ -187,13 +152,13 @@ module.exports = (app) => {
}

async function insertIntoFile(context) {
let fileContent = "";
let results = [];
let newContent = "";
let resultString = "";
let results = [];

try {

fileContent = await GetSurveyData(context);
newContent = await GetSurveyData(context);

// Try to get the file
let file = await context.octokit.repos.getContent({
Expand All @@ -207,6 +172,7 @@ module.exports = (app) => {
let fileContents = Buffer.from(file.data.content, "base64").toString();
// If the file contents are not empty, parse the CSV
if (fileContents.length > 0) {
app.log.info("Starting to parse the CSV file...");
// create a readable stream
let readableStream = new stream.Readable();
readableStream.push(fileContents);
Expand All @@ -232,14 +198,14 @@ module.exports = (app) => {
if(issue_id_index != -1){
// save previous comments
if (results[issue_id_index].comment) {
fileContent.comment = results[issue_id_index].comment + ' || ' + fileContent.comment;
newContent.comment = results[issue_id_index].comment + ' || ' + newContent.comment;
}

// if the issue_id exists, update the row in the array results
results[issue_id_index] = fileContent;
results[issue_id_index] = newContent;
}else{
// if the issue_id does not exist, push the row into the array
results.push(fileContent);
results.push(newContent);
}

resultString = Object.keys(results[0]).join(',') + '\n';
Expand Down Expand Up @@ -267,8 +233,8 @@ module.exports = (app) => {
} catch (error) {
// If the file does not exist, create it
if (error.status === 404) {
let completeData = 'enterprise_name,organization_name,repository_name,issue_id,issue_number,PR_number,assignee_name,is_copilot_used,saving_percentage,usage_frequency,comment,created_at,completed_at\n'
+ Object.values(fileContent).join(',');
let completeData = 'enterprise_name,organization_name,repository_name,issue_id,issue_number,PR_number,assignee_name,is_copilot_used,saving_percentage,usage_frequency,savings_invested,comment,created_at,completed_at\n'
+ Object.values(newContent).join(',');
await createBranch(context);
await context.octokit.repos.createOrUpdateFileContents({
owner: context.payload.repository.owner.login,
Expand All @@ -282,49 +248,64 @@ module.exports = (app) => {
app.log.error(error);
}
}

async function createBranch(context) {
// Step 1: Get reference to the default branch
let ref;
try {
// Try to get the 'main' branch
const { data: mainRef } = await context.octokit.git.getRef({
owner: context.payload.repository.owner.login,
repo: context.payload.repository.name,
ref: 'heads/main',
});
ref = mainRef;
} catch (error) {
// If 'main' branch does not exist, try to get the 'master' branch
if (error.status === 404) {
const { data: masterRef } = await context.octokit.git.getRef({
owner: context.payload.repository.owner.login,
repo: context.payload.repository.name,
ref: 'heads/master',
});
ref = masterRef;
} else {
app.log.error(error);
}
}

// Step 2: Create a new branch from the default branch
try {
await context.octokit.git.createRef({
owner: context.payload.repository.owner.login,
repo: context.payload.repository.name,
ref: `refs/heads/${BranchName}`,
sha: ref.object.sha,
});
} catch (error) {
if (error.status === 422) {
app.log.info(`Branch ${BranchName} already exists`);
} else {
app.log.error(error);
}
}
}

async function createBranch(context) {
// Step 1: Get reference to the default branch
let RefBranch = null;
try {
// Try to get the 'main' branch
RefBranch = await context.octokit.git.getRef({
owner: context.payload.repository.owner.login,
repo: context.payload.repository.name,
ref: 'heads/main',
});
} catch (error) {
// If 'main' branch does not exist, try to get the 'master' branch
if (error.status === 404) {
RefBranch = await context.octokit.git.getRef({
owner: context.payload.repository.owner.login,
repo: context.payload.repository.name,
ref: 'heads/master',
});
} else {
app.log.error(error);
}
}

// Step 2: Create a new branch from the default branch
try {
await context.octokit.git.createRef({
owner: context.payload.repository.owner.login,
repo: context.payload.repository.name,
ref: `refs/heads/${BranchName}`,
sha: RefBranch.data.object.sha,
});
} catch (error) {
if (error.status === 422) {
app.log.info(`Branch ${BranchName} already exists`);
} else {
app.log.error(error);
}
}
}

async function getQuestionResponse(start, end, issue_body) {
app.log.info("Getting answers for question " + start + " to " + end);
let AnswerSelected = false;
let Answers = new Array();
let Expression = end ? new RegExp(start + "\\. (.*" + end + "\\." + ")?", "s") : new RegExp(start + "\\. (.*" + ")?", "s");
let QuestionOptions = issue_body.match(Expression)[0].match(/\[x\].*/g);
if(QuestionOptions){
AnswerSelected = true;
QuestionOptions.forEach((option) => {
let cleanAnswer = option;
cleanAnswer = cleanAnswer.replace(/\[x\] /g, "");
Answers.push(cleanAnswer);
});
}
return AnswerSelected ? Answers : null;
}

// For more information on building apps:
Expand Down
9 changes: 8 additions & 1 deletion issue_template/copilot-usage-en.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,11 @@ For Pull Request XXX:

5. ***What other information can you share about Copilot's ability to save you time coding?***

- (Please tell us in a comment)
- (Please tell us in a comment)

6. ***Where did you invest your Copilot Time Savings?***
- [ ] Resolve vulnerabilites
- [ ] Experiment + Learn and Wellness
- [ ] Technical debt and refactorization
- [ ] Work on other items in the backlog
- [ ] Other. Please explain in the comment
9 changes: 8 additions & 1 deletion issue_template/copilot-usage-es.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,11 @@ Para el Pull Request XXX:

5. ***¿Qué otra información puedes compartir sobre la capacidad de Copilot para ahorrarte tiempo codificando?***

- (Por favor díganos en un comentario)
- (Por favor díganos en un comentario)

6. ***¿En qué has invertido el tiempo ganado con Copilot?***
- [ ] Resolver vulnerabilidades
- [ ] Experimentar + Aprender y Bienestar
- [ ] Reducir deuda técnica y/o refactorización
- [ ] Trabajar en otras tareas en el backlog
- [ ] Otros. Por favor explica en el comentario
9 changes: 8 additions & 1 deletion issue_template/copilot-usage-fr.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,11 @@ Pour le Pull Request XXX :

5. ***Quelles autres informations pouvez-vous partager sur la capacité de Copilot à vous faire gagner du temps en matière de codage ?***

- (Merci de nous le dire dans un commentaire)
- (Merci de nous le dire dans un commentaire)

6. ***Qu'avez-vous investi dans le temps gagné avec Copilot ?***
- [ ] Résoudre les vulnérabilités
- [ ] Expérience + Apprendre et/ou Bien-être
- [ ] Réduire la dette technique et/ou refactoring
- [ ] Travailler sur d'autres tâches dans le backlog
- [ ] Autres. Veuillez expliquer dans le commentaire
Loading

0 comments on commit d8bdd82

Please sign in to comment.