Skip to content

Solution based merging for data collector#1676

Open
daveMueller wants to merge 22 commits intocoverlet-coverage:masterfrom
daveMueller:1307_solution-based-merging-collector
Open

Solution based merging for data collector#1676
daveMueller wants to merge 22 commits intocoverlet-coverage:masterfrom
daveMueller:1307_solution-based-merging-collector

Conversation

@daveMueller
Copy link
Copy Markdown
Collaborator

closes #1307

@daveMueller
Copy link
Copy Markdown
Collaborator Author

I guess I need some additional feedback/help here. The main problem is that I still don't fully understand how it is intended to implement IDataCollectorAttachmentProcessor. I see it beeing executed for every test project where the attachment then only is the projects test result. And also it gets executed once for the whole solution where the attachments are all the test runs results. Thus I could only check if attachments > 1 and then merge all of them together.
But from previous discussions it seems like there could be intermediate results that need to merged.
Therfore I added now a new runsettings option called MergeDirectory to store intermediate results. The idea behind that is that everytime the attachment processor triggers all attachments are merged plus the intermediate result in the MergeDirectory. The result of this merge is the new intermediate result.
Unfortunately the MergeDirectory path must be an absolute path so that this works reliable. If it is a relative path, the location changes depending on the if the processor is triggered from a project or from a solution.
cc: @MarcoRossignoli @bert2

@MarcoRossignoli
Copy link
Copy Markdown
Collaborator

MarcoRossignoli commented Aug 13, 2024

Therfore I added now a new runsettings option called MergeDirectory to store intermediate results. The idea behind that is that everytime the attachment processor triggers all attachments are merged plus the intermediate result in the MergeDirectory. The result of this merge is the new intermediate result.

Why you need intermediate? take the directory from the first of the 2 attachments you receive and save in the same directory removing the 2 and send back. Something like a reduce. Attachments usually are inside the result directory specified in the config.

@daveMueller daveMueller marked this pull request as ready for review March 4, 2025 23:08
@Bertk Bertk requested a review from Copilot June 9, 2025 13:40
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements solution based merging for the data collector by introducing a new ReportMerging option. Key changes include:

  • Adding a new ReportMerging flag and parsing logic via the ReportFormatParser.
  • Refactoring settings parsing and reporter creation to include JSON reports when merging is enabled.
  • Implementing a new post processor (CoverletCoveragePostProcessor) for merging multiple coverage reports and updating documentation accordingly.

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/coverlet.core/Helpers/SourceRootTranslator.cs Added an additional constructor initializing mapping dictionaries.
src/coverlet.collector/Utilities/ReportFormatParser.cs Introduced a new helper for parsing report formats and related configuration elements.
src/coverlet.collector/Utilities/CoverletConstants.cs Added a new constant for the ReportMerging option.
src/coverlet.collector/DataCollection/CoverletSettingsParser.cs Refactored to use ReportFormatParser and parse the new ReportMerging flag.
src/coverlet.collector/DataCollection/CoverletSettings.cs Added the new ReportMerging property and updated ToString output.
src/coverlet.collector/DataCollection/CoverletCoverageCollector.cs Added a new attribute to integrate the post processor.
src/coverlet.collector/DataCollection/CoverageManager.cs Modified reporter creation to automatically include the JSON reporter when report merging is enabled.
src/coverlet.collector/ArtifactPostProcessor/CoverletCoveragePostProcessor.cs Added a new post processor implementing the merging logic for coverage reports.
Documentation/VSTestIntegration.md Updated runsettings documentation to include the ReportMerging option.
Documentation/Troubleshooting.md Added guidance for debugging the post processor.
Documentation/Changelog.md Documented the improvements and new merging feature.

Comment on lines +37 to +40
if (settings.ReportMerging && ! settings.ReportFormats.Contains("json"))
settings.ReportFormats = settings.ReportFormats.Append("json").ToArray();

return settings.ReportFormats.Select(format =>
Copy link

Copilot AI Jun 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mutating settings.ReportFormats in place may introduce unintended side effects; consider assigning the updated report formats to a new local variable to preserve immutability.

Suggested change
if (settings.ReportMerging && ! settings.ReportFormats.Contains("json"))
settings.ReportFormats = settings.ReportFormats.Append("json").ToArray();
return settings.ReportFormats.Select(format =>
IReporter[] updatedReportFormats = settings.ReportFormats;
if (settings.ReportMerging && ! updatedReportFormats.Contains("json"))
updatedReportFormats = updatedReportFormats.Append("json").ToArray();
return updatedReportFormats.Select(format =>

Copilot uses AI. Check for mistakes.
Comment on lines +78 to +79
if (! string.IsNullOrEmpty(directory) && Directory.Exists(directory))
Directory.Delete(directory, true);
Copy link

Copilot AI Jun 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding checks or logging to ensure that deleting the directory does not inadvertently remove non-report files; additional filtering criteria may improve safety.

Suggested change
if (! string.IsNullOrEmpty(directory) && Directory.Exists(directory))
Directory.Delete(directory, true);
if (!string.IsNullOrEmpty(directory) && Directory.Exists(directory))
{
var files = Directory.GetFiles(directory);
var reportFiles = files.Where(file => file.EndsWith(".json") || file.EndsWith(".xml")).ToList();
if (reportFiles.Count == files.Length) // Ensure all files are report-related
{
Console.WriteLine($"Deleting directory: {directory}. Files: {string.Join(", ", reportFiles)}");
Directory.Delete(directory, true);
}
else
{
Console.WriteLine($"Skipping directory: {directory}. Contains non-report files.");
}
}

Copilot uses AI. Check for mistakes.
Comment on lines +29 to +38
_sourceRootMapping = new Dictionary<string, List<SourceRootMapping>>();
_sourceToDeterministicPathMapping = new Dictionary<string, List<string>>();
}

public SourceRootTranslator(ILogger logger, IFileSystem fileSystem)
{
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_fileSystem = fileSystem ?? throw new ArgumentNullException(nameof(fileSystem));
_sourceRootMapping = new Dictionary<string, List<SourceRootMapping>>();
_sourceToDeterministicPathMapping = new Dictionary<string, List<string>>();
Copy link

Copilot AI Jun 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] Since similar dictionary initializations are repeated across constructors, consider consolidating them into a common initializer to reduce duplication.

Suggested change
_sourceRootMapping = new Dictionary<string, List<SourceRootMapping>>();
_sourceToDeterministicPathMapping = new Dictionary<string, List<string>>();
}
public SourceRootTranslator(ILogger logger, IFileSystem fileSystem)
{
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_fileSystem = fileSystem ?? throw new ArgumentNullException(nameof(fileSystem));
_sourceRootMapping = new Dictionary<string, List<SourceRootMapping>>();
_sourceToDeterministicPathMapping = new Dictionary<string, List<string>>();
InitializeMappings();
}
public SourceRootTranslator(ILogger logger, IFileSystem fileSystem)
{
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_fileSystem = fileSystem ?? throw new ArgumentNullException(nameof(fileSystem));
InitializeMappings();

Copilot uses AI. Check for mistakes.
@Bertk Bertk added the feature PR label for new features label Dec 9, 2025
@Bertk Bertk added driver-collectors Issue related to collectors driver coverlet-core labels Jan 22, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 14 comments.

Comment on lines +34 to +36
public Task<ICollection<AttachmentSet>> ProcessAttachmentSetsAsync(XmlElement configurationElement,
ICollection<AttachmentSet> attachments, IProgress<int> progressReporter,
IMessageLogger logger, CancellationToken cancellationToken)
Copy link

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cancellationToken parameter is not used anywhere in the method. While the current implementation uses synchronous operations, the method signature suggests async cancellation support. Consider either utilizing the cancellationToken for long-running operations (like file I/O, JSON deserialization, and directory deletion), or document why it's not needed. At minimum, cancellationToken.ThrowIfCancellationRequested() should be called before expensive operations.

Copilot uses AI. Check for mistakes.

private static bool IsFileWithJsonExt(UriDataAttachment x)
{
return IsFileAttachment(x) && Path.GetExtension(x.Uri.AbsolutePath).Equals(".json");
Copy link

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

String comparison is case-sensitive. The extension comparison with ".json" won't match files with uppercase extensions like ".JSON" or ".Json". Consider using Path.GetExtension(x.Uri.AbsolutePath).Equals(".json", StringComparison.OrdinalIgnoreCase) for case-insensitive comparison.

Suggested change
return IsFileAttachment(x) && Path.GetExtension(x.Uri.AbsolutePath).Equals(".json");
return IsFileAttachment(x) && Path.GetExtension(x.Uri.AbsolutePath).Equals(".json", StringComparison.OrdinalIgnoreCase);

Copilot uses AI. Check for mistakes.
Comment on lines +38 to +41
_reportFormatParser ??= new ReportFormatParser();
_coverageResult ??= new CoverageResult();
_coverageResult.Modules ??= new Modules();
_logger = logger;
Copy link

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thread safety concern: The class uses instance fields (_coverageResult, _reportFormatParser, _logger) that are initialized with null-coalescing operators and potentially shared across multiple calls to ProcessAttachmentSetsAsync. If this method is called concurrently or multiple times, there's potential for race conditions or state corruption. Since SupportsIncrementalProcessing is set to true (line 30), the class instance may be reused. Consider whether these fields should be method-local variables instead, or document thread-safety guarantees.

Copilot uses AI. Check for mistakes.

private static IReporter[] CreateReporters(CoverletSettings settings, TestPlatformEqtTrace eqtTrace)
{
if (settings.ReportMerging && ! settings.ReportFormats.Contains("json"))
Copy link

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spacing inconsistency: there's an extra space after the exclamation mark in this condition. Consider removing it for consistency with standard formatting conventions (e.g., !settings.ReportFormats.Contains("json") instead of ! settings.ReportFormats.Contains("json")).

Suggested change
if (settings.ReportMerging && ! settings.ReportFormats.Contains("json"))
if (settings.ReportMerging && !settings.ReportFormats.Contains("json"))

Copilot uses AI. Check for mistakes.
fileAttachments.ForEach(x =>
{
string directory = Path.GetDirectoryName(x.Uri.LocalPath);
if (! string.IsNullOrEmpty(directory) && Directory.Exists(directory))
Copy link

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spacing inconsistency: there's an extra space after the exclamation mark in this condition. Consider removing it for consistency with standard formatting conventions (e.g., !string.IsNullOrEmpty(directory) instead of ! string.IsNullOrEmpty(directory)).

Suggested change
if (! string.IsNullOrEmpty(directory) && Directory.Exists(directory))
if (!string.IsNullOrEmpty(directory) && Directory.Exists(directory))

Copilot uses AI. Check for mistakes.

if (!reportMerging) return Task.FromResult(attachments);

IList<IReporter> reporters = CreateReporters(formats).ToList();
Copy link

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If all report formats are invalid, the reporters list will be empty, but the code proceeds to use it without checking. When WriteCoverageReports is called with an empty reporters list (line 65), it will create an empty AttachmentSet, which may not be the desired behavior. Consider checking if the reporters list is empty and either logging a warning or returning early.

Copilot uses AI. Check for mistakes.
Comment on lines +59 to +60
string mergeFilePath = Path.GetDirectoryName(fileAttachments.First().Uri.LocalPath);

Copy link

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential bug: calling First() on an empty collection will throw an InvalidOperationException. If there are no file attachments in the collection, this will cause a runtime error. Consider adding a check to ensure fileAttachments is not empty before calling First(), or use FirstOrDefault() with a null check.

Suggested change
string mergeFilePath = Path.GetDirectoryName(fileAttachments.First().Uri.LocalPath);
var firstFileAttachment = fileAttachments.FirstOrDefault();
if (firstFileAttachment == null)
{
_logger?.SendMessage(TestMessageLevel.Warning, "No file attachments found to merge coverage reports.");
return Task.FromResult(attachments);
}
string mergeFilePath = Path.GetDirectoryName(firstFileAttachment.Uri.LocalPath);

Copilot uses AI. Check for mistakes.
Comment on lines +27 to +31
public SourceRootTranslator()
{
_sourceRootMapping = new Dictionary<string, List<SourceRootMapping>>();
_sourceToDeterministicPathMapping = new Dictionary<string, List<string>>();
}
Copy link

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parameterless constructor leaves the _logger and _fileSystem fields as null. While the comment on line 127 states that "empty source root translator returns the original path," methods like ResolveFilePath (line 147) will attempt to access _fileSystem which could cause NullReferenceException. Either document that the parameterless constructor is only meant for limited use cases where these methods won't be called, or ensure all methods handle null fields gracefully.

Copilot uses AI. Check for mistakes.
Comment on lines +119 to +120
string json = File.ReadAllText(filePath);
coverageResult.Merge(JsonConvert.DeserializeObject<Modules>(json));
Copy link

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing error handling: JSON deserialization can fail for various reasons (malformed JSON, incompatible schema, file read errors). If deserialization fails, it will throw an exception that propagates up without a meaningful error message. Consider wrapping this in a try-catch block and providing a more descriptive error message that includes the file path.

Suggested change
string json = File.ReadAllText(filePath);
coverageResult.Merge(JsonConvert.DeserializeObject<Modules>(json));
try
{
string json = File.ReadAllText(filePath);
Modules modules = JsonConvert.DeserializeObject<Modules>(json);
coverageResult.Merge(modules);
}
catch (Exception ex)
{
throw new CoverletDataCollectorException(
$"{CoverletConstants.DataCollectorName}: Failed to merge coverage result from file '{filePath}'",
ex);
}

Copilot uses AI. Check for mistakes.
Comment on lines +1 to +162
// Copyright (c) Toni Solarin-Sodara
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using System.Xml;
using Coverlet.Collector.Utilities;
using Coverlet.Core;
using Coverlet.Core.Abstractions;
using Coverlet.Core.Reporters;
using Microsoft.VisualStudio.TestPlatform.ObjectModel;
using Microsoft.VisualStudio.TestPlatform.ObjectModel.DataCollection;
using Microsoft.VisualStudio.TestPlatform.ObjectModel.Logging;
using Newtonsoft.Json;
using System.Diagnostics;
using Coverlet.Core.Helpers;

namespace coverlet.collector.ArtifactPostProcessor
{
public class CoverletCoveragePostProcessor : IDataCollectorAttachmentProcessor
{
private CoverageResult _coverageResult;
private ReportFormatParser _reportFormatParser;
private IMessageLogger _logger;

public bool SupportsIncrementalProcessing => true;

public IEnumerable<Uri> GetExtensionUris() => new[] { new Uri(CoverletConstants.DefaultUri) };

public Task<ICollection<AttachmentSet>> ProcessAttachmentSetsAsync(XmlElement configurationElement,
ICollection<AttachmentSet> attachments, IProgress<int> progressReporter,
IMessageLogger logger, CancellationToken cancellationToken)
{
_reportFormatParser ??= new ReportFormatParser();
_coverageResult ??= new CoverageResult();
_coverageResult.Modules ??= new Modules();
_logger = logger;

string[] formats = _reportFormatParser.ParseReportFormats(configurationElement);
bool deterministic = _reportFormatParser.ParseDeterministicReport(configurationElement);
bool useSourceLink = _reportFormatParser.ParseUseSourceLink(configurationElement);
bool reportMerging = _reportFormatParser.ParseReportMerging(configurationElement);

AttachDebugger();

if (!reportMerging) return Task.FromResult(attachments);

IList<IReporter> reporters = CreateReporters(formats).ToList();

if (attachments.Count > 1)
{
_coverageResult.Parameters = new CoverageParameters() {DeterministicReport = deterministic, UseSourceLink = useSourceLink };

var fileAttachments = attachments.SelectMany(x => x.Attachments.Where(IsFileAttachment)).ToList();
string mergeFilePath = Path.GetDirectoryName(fileAttachments.First().Uri.LocalPath);

MergeExistingJsonReports(attachments);

RemoveObsoleteReports(fileAttachments);

AttachmentSet mergedFileAttachment = WriteCoverageReports(reporters, mergeFilePath, _coverageResult);

attachments = new List<AttachmentSet> { mergedFileAttachment };
}

return Task.FromResult(attachments);
}

private static void RemoveObsoleteReports(List<UriDataAttachment> fileAttachments)
{
fileAttachments.ForEach(x =>
{
string directory = Path.GetDirectoryName(x.Uri.LocalPath);
if (! string.IsNullOrEmpty(directory) && Directory.Exists(directory))
Directory.Delete(directory, true);
});
}

private void MergeExistingJsonReports(IEnumerable<AttachmentSet> attachments)
{
foreach (AttachmentSet attachmentSet in attachments)
{
attachmentSet.Attachments.Where(IsFileWithJsonExt).ToList().ForEach(x =>
MergeWithCoverageResult(x.Uri.LocalPath, _coverageResult)
);
}
}

private AttachmentSet WriteCoverageReports(IEnumerable<IReporter> reporters, string directory, CoverageResult coverageResult)
{
var attachment = new AttachmentSet(new Uri(CoverletConstants.DefaultUri), string.Empty);
foreach (IReporter reporter in reporters)
{
string report = GetCoverageReport(coverageResult, reporter);
var file = new FileInfo(Path.Combine(directory, Path.ChangeExtension(CoverletConstants.DefaultFileName, reporter.Extension)));
file.Directory?.Create();
File.WriteAllText(file.FullName, report);
attachment.Attachments.Add(new UriDataAttachment(new Uri(file.FullName),string.Empty));
}
return attachment;
}

private static bool IsFileWithJsonExt(UriDataAttachment x)
{
return IsFileAttachment(x) && Path.GetExtension(x.Uri.AbsolutePath).Equals(".json");
}

private static bool IsFileAttachment(UriDataAttachment x)
{
return x.Uri.IsFile;
}

private void MergeWithCoverageResult(string filePath, CoverageResult coverageResult)
{
string json = File.ReadAllText(filePath);
coverageResult.Merge(JsonConvert.DeserializeObject<Modules>(json));
}

private string GetCoverageReport(CoverageResult coverageResult, IReporter reporter)
{
try
{
// empty source root translator returns the original path for deterministic report
return reporter.Report(coverageResult, new SourceRootTranslator());
}
catch (Exception ex)
{
throw new CoverletDataCollectorException(
$"{CoverletConstants.DataCollectorName}: Failed to get coverage report", ex);
}
}

private void AttachDebugger()
{
if (int.TryParse(Environment.GetEnvironmentVariable("COVERLET_DATACOLLECTOR_POSTPROCESSOR_DEBUG"), out int result) && result == 1)
{
Debugger.Launch();
Debugger.Break();
}
}

private IEnumerable<IReporter> CreateReporters(IEnumerable<string> formats)
{
IEnumerable<IReporter> reporters = formats.Select(format =>
{
var reporterFactory = new ReporterFactory(format);
if (!reporterFactory.IsValidFormat())
{
_logger.SendMessage(TestMessageLevel.Warning, $"Invalid report format '{format}'");
return null;
}
return reporterFactory.CreateReporter();
}).Where(r => r != null);

return reporters;
}
}
}
Copy link

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new CoverletCoveragePostProcessor class, which is the core of this feature, lacks test coverage. Given the complexity of the attachment processing logic (merging JSON reports, handling file deletions, creating merged attachments), this class should have comprehensive unit tests to cover various scenarios such as: multiple attachments, empty attachments, malformed JSON files, missing directories, and the interaction with the merging flag.

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

coverlet-core driver-collectors Issue related to collectors driver feature PR label for new features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement solution based merging for data collector

5 participants