Add linguist tags processor

Signed-off-by: Brian Forbis <bforbis@athenahealth.com>

.changeset/large-badgers-switch.md

@@ -0,0 +1,5 @@
+---
+'@backstage/plugin-linguist-common': patch
+---
+
+Exported new LanguageType type alias

.changeset/strange-shrimps-mix.md

@@ -0,0 +1,5 @@
+---
+'@backstage/plugin-linguist-backend': minor
+---
+
+Adds a processor to the linguist backend which can automatically add language tags to entities

plugins/linguist-backend/README.md

@@ -1,6 +1,6 @@
 # Linguist Backend
 
-Welcome to the Linguist backend plugin! This plugin provides data for the Linguist frontend features.
+Welcome to the Linguist backend plugin! This plugin provides data for the Linguist frontend features. Additionally, it provides an optional entity processor which will automate adding language tags to your entities.
 
 ## Setup
 
@@ -144,6 +144,130 @@ return createRouter(
 
 **Note:** This has the potential to cause a lot of processing, be very thoughtful about this before hand
 
+## Linguist Tags Processor
+
+The `LinguistTagsProcessor` can be added into your catalog builder as a way to incorporate the language breakdown from linguist as `metadata.tags` on your entities. Doing so enables the ability to easily filter for entities in your catalog index based on the language of the source repository.
+
+### Processor Setup
+
+Setup the linguist tag processor in `packages/backend/src/plugins/catalog.ts`.
+
+```ts
+import { LinguistTagsProcessor } from '@backstage/plugin-linguist-backend';
+// ...
+export default async function createPlugin(
+  // ...
+  builder.addProcessor(
+    LinguistTagsProcessor.fromConfig(env.config, {
+      logger: env.logger,
+      discovery: env.discovery,
+    })
+  );
+```
+
+### Processor Options
+
+The processor accepts configurations either directly as options when constructing using `fromConfig()`, or can also be configured in `app-config.yaml` with the same fields.
+
+Example linguist processor configuration:
+
+```yaml
+linguist:
+  tagsProcessor:
+    bytesThreshold: 1000
+    languageTypes: ['programming', 'markup']
+    languageMap:
+      Dockerfile: ''
+      TSX: 'react'
+    cacheTTL:
+      hours: 24
+```
+
+#### `languageMap`
+
+The `languageMap` option allows you to build a custom map of linguist languages to how you want them to show up as tags. The keys should be exact matches to languages in the [linguist dataset](https://github.com/github-linguist/linguist/blob/master/lib/linguist/languages.yml) and the values should be how they render as backstage tags. These values will be used "as is" and will not be further transformed.
+
+Keep in mind that backstage has [character requirements for tags](https://backstage.io/docs/features/software-catalog/descriptor-format#tags-optional). If your map emits an invalid tag, it will cause an error during processing and your entity will not be processed.
+
+If you map a key to `''`, it will not be emitted as a tag. This can be useful if you want to ignore some of the linguist languages.
+
+```yaml
+linguist:
+  tagsProcessor:
+    languageMap:
+      # You don't want dockerfile to show up as a tag
+      Dockerfile: ''
+      # Be more specific about what the file is
+      HCL: terraform
+      # A more casual tag for a formal name
+      Protocol Buffer: protobuf
+```
+
+#### `cacheTTL`
+
+The `cacheTTL` option allows you to determine for how long this processor will cache languages for an `entityRef` before refreshing from the linguist backend. As this processor will run continuously, this cache is supplied to limit the load done on the linguist DB and API.
+
+By default, this processor will cache languages for 30 minutes before refreshing from the linguist database.
+
+You can optionally disable the cache entirely by passing in a `cacheTTL` duration of 0 minutes.
+
+```yaml
+linguist:
+  tagsProcessor:
+    cacheTTL: { minutes: 0 }
+```
+
+#### `bytesThreshold`
+
+The `bytesThreshold` option allows you to control a number of bytes threshold which must be surpassed before a language tag will be emitted by this processor. As an example, some repositories may have short build scripts written in Bash, but you may only want the main language of the project emitted (an alternate way to control this is to use the `languageMap` to map `Shell` languages to `undefined`).
+
+```yaml
+linguist:
+  tagsProcessor:
+    # Ignore languages with less than 5000 bytes in a repo.
+    bytesThreshold: 5000
+```
+
+#### `languageTypes`
+
+The `languageTypes` option allows you to control what categories of linguist languages are automatically added as tags. By default, this will only include language tags of type `programming`, but you can pass in a custom array here to allow adding other language types.
+
+You can see the full breakdown of linguist supported languages [in their repo](https://github.com/github-linguist/linguist/blob/master/lib/linguist/languages.yml).
+
+For example, you may want to also include languages of type `data`
+
+```yaml
+linguist:
+  tagsProcessor:
+    languageTypes:
+      - programming
+      - data
+```
+
+#### `shouldProcessEntity`
+
+The `shouldProcessEntity` is a function you can pass into the processor which determines which entities should have language tags fetched from linguist and added to the entity. By default, this will only run on entities of `kind: Component`, however this function let's you fully customize which entities should be processed.
+
+As an example, you may choose to extend this to support both `Component` and `Resource` kinds along with allowing an opt-in annotation on the entity which entity authors can use.
+
+As this option is a function, it cannot be configured in `app-config.yaml`. You must pass this as an option within typescript.
+
+```ts
+LinguistLanguageTagsProcessor.fromConfig(env.config, {
+  logger: env.logger,
+  discovery: env.discovery,
+  shouldProcessEntity: (entity: Entity) => {
+    if (
+      ['Component', 'Resource'].includes(entity.kind) &&
+      entity.metadata.annotations?.['some-custom-annotation']
+    ) {
+      return true;
+    }
+    return false;
+  },
+});
+```
+
 ## Links
 
 - [Frontend part of the plugin](https://github.com/backstage/backstage/tree/master/plugins/linguist)

plugins/linguist-backend/api-report.md

@@ -4,9 +4,15 @@
 
 ```ts
 import { BackendFeature } from '@backstage/backend-plugin-api';
+import { CatalogProcessor } from '@backstage/plugin-catalog-node';
+import { CatalogProcessorCache } from '@backstage/plugin-catalog-node';
+import { Config } from '@backstage/config';
+import { DiscoveryService } from '@backstage/backend-plugin-api';
+import { Entity } from '@backstage/catalog-model';
 import express from 'express';
 import { HumanDuration } from '@backstage/types';
 import { Languages } from '@backstage/plugin-linguist-common';
+import { LanguageType } from '@backstage/plugin-linguist-common';
 import { Logger } from 'winston';
 import { PluginDatabaseManager } from '@backstage/backend-common';
 import { PluginEndpointDiscovery } from '@backstage/backend-common';
@@ -48,6 +54,38 @@ export interface LinguistPluginOptions {
   useSourceLocation?: boolean;
 }
 
+// @public
+export class LinguistTagsProcessor implements CatalogProcessor {
+  constructor(options: LinguistTagsProcessorOptions);
+  // (undocumented)
+  static fromConfig(
+    config: Config,
+    options: LinguistTagsProcessorOptions,
+  ): LinguistTagsProcessor;
+  // (undocumented)
+  getProcessorName(): string;
+  preProcessEntity(
+    entity: Entity,
+    _: any,
+    __: any,
+    ___: any,
+    cache: CatalogProcessorCache,
+  ): Promise<Entity>;
+}
+
+// @public
+export interface LinguistTagsProcessorOptions {
+  bytesThreshold?: number;
+  cacheTTL?: HumanDuration;
+  // (undocumented)
+  discovery: DiscoveryService;
+  languageMap?: Record<string, string | undefined>;
+  languageTypes?: LanguageType[];
+  // (undocumented)
+  logger: Logger;
+  shouldProcessEntity?: ShouldProcessEntity;
+}
+
 // @public (undocumented)
 export interface PluginOptions {
   // (undocumented)
@@ -81,4 +119,7 @@ export interface RouterOptions {
   // (undocumented)
   tokenManager: TokenManager;
 }
+
+// @public
+export type ShouldProcessEntity = (entity: Entity) => boolean;
 ```

plugins/linguist-backend/config.d.ts

@@ -0,0 +1,46 @@
+/*
+ * Copyright 2023 The Backstage Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { HumanDuration } from '@backstage/types';
+
+export interface Config {
+  /** Configuration options for the linguist plugin */
+  linguist?: {
+    /** Options for the tags processor */
+    tagsProcessor?: {
+      /**
+       * Determines how many bytes of a language should be in a repo
+       * for it to be added as an entity tag. Defaults to 0.
+       */
+      bytesThreshold?: number;
+      /**
+       * The types of linguist languages that should be processed. Can be
+       * any of "programming", "data", "markup", "prose". Defaults to ["programming"].
+       */
+      languageTypes?: string[];
+      /**
+       * A custom mapping of linguist languages to how they should be rendered as entity tags.
+       * If a language is mapped to '' it will not be included as a tag.
+       */
+      languageMap?: Record<string, string | undefined>;
+      /**
+       * How long to cache entity languages for in memory. Used to avoid constant db hits during
+       * processing. Defaults to 30 minutes.
+       */
+      cacheTTL?: HumanDuration;
+    };
+  };
+}

plugins/linguist-backend/package.json

@@ -30,6 +30,7 @@
     "@backstage/config": "workspace:^",
     "@backstage/errors": "workspace:^",
     "@backstage/plugin-auth-node": "workspace:^",
+    "@backstage/plugin-catalog-node": "workspace:^",
     "@backstage/plugin-linguist-common": "workspace:^",
     "@backstage/types": "workspace:^",
     "@types/express": "*",
@@ -48,11 +49,14 @@
     "@backstage/backend-test-utils": "workspace:^",
     "@backstage/cli": "workspace:^",
     "@types/supertest": "^2.0.8",
+    "js-yaml": "^4.1.0",
     "msw": "^1.0.0",
     "supertest": "^6.2.4"
   },
   "files": [
     "dist",
+    "config.d.ts",
     "migrations/**/*.{js,d.ts}"
-  ]
+  ],
+  "configSchema": "config.d.ts"
 }

plugins/linguist-backend/src/index.ts

@@ -20,6 +20,7 @@
  * @packageDocumentation
  */
 
+export * from './processor';
 export * from './service/router';
 export type { LinguistBackendApi } from './api';
 export { linguistPlugin } from './plugin';

plugins/linguist-backend/src/processor/LinguistTagsProcessor.test.ts

@@ -0,0 +1,343 @@
+/*
+ * Copyright 2023 The Backstage Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import {
+  LinguistTagsProcessor,
+  LinguistTagsProcessorOptions,
+  sanitizeTag,
+} from './LinguistTagsProcessor';
+import { ConfigReader } from '@backstage/config';
+import { getVoidLogger } from '@backstage/backend-common';
+import { CatalogProcessorCache } from '@backstage/plugin-catalog-node';
+import { Entity, makeValidator } from '@backstage/catalog-model';
+import { DiscoveryService } from '@backstage/backend-plugin-api';
+import fetch, { Response } from 'node-fetch';
+import * as path from 'path';
+import yaml from 'js-yaml';
+import * as fs from 'fs';
+
+const { isValidTag } = makeValidator();
+
+jest.mock('node-fetch', () => jest.fn());
+const mockedFetch: jest.MockedFunction<typeof fetch> =
+  fetch as jest.MockedFunction<typeof fetch>;
+
+const discovery: DiscoveryService = {
+  getBaseUrl: jest.fn().mockResolvedValue('http://example.com/api/linguist'),
+  getExternalBaseUrl: jest.fn(),
+};
+
+let state: Record<string, any> = {};
+const mockCacheGet = jest
+  .fn()
+  .mockImplementation(async (key: string) => state[key]);
+const mockCacheSet = jest.fn().mockImplementation((key: string, value: any) => {
+  state[key] = value;
+});
+const cache: CatalogProcessorCache = {
+  get: mockCacheGet,
+  set: mockCacheSet,
+};
+
+describe('sanitizeTag', () => {
+  const linguistDataSet = yaml.load(
+    fs.readFileSync(
+      path.resolve(require.resolve('linguist-js'), '../../ext/languages.yml'),
+      'utf-8',
+    ),
+  ) as Object;
+  const languages = Object.keys(linguistDataSet);
+  test('Should clean up all linguist languages', () => {
+    const invalid = languages
+      .map(sanitizeTag)
+      .filter(lang => !isValidTag(lang));
+    expect(invalid).toStrictEqual([]);
+    // Keep a snapshot here so that as new languages are added to linguist,
+    // we can spot check them to make sure the transformer for them makes sense.
+    expect(languages.map(sanitizeTag)).toMatchSnapshot();
+  });
+});
+
+describe('LinguistTagsProcessor', () => {
+  afterEach(() => {
+    mockedFetch.mockReset();
+    mockCacheGet.mockClear();
+    mockCacheSet.mockClear();
+    state = {};
+  });
+
+  test('Should construct fromConfig', () => {
+    const config = new ConfigReader({
+      linguist: {},
+    });
+    expect(() => {
+      return LinguistTagsProcessor.fromConfig(config, {
+        logger: getVoidLogger(),
+        discovery,
+      });
+    }).not.toThrow();
+  });
+
+  test('Should assign valid language tags', async () => {
+    const processor = buildProcessor({});
+
+    mockFetchImplementation();
+    const entity = baseEntity();
+    await processor.preProcessEntity(entity, null, null, null, cache);
+    expect(mockedFetch).toHaveBeenCalledTimes(1);
+    expect(entity.metadata.tags).toStrictEqual([
+      'c++',
+      'asp-dot-net',
+      'java',
+      'common-lisp',
+    ]);
+
+    entity.metadata.tags?.forEach(tag => {
+      expect(isValidTag(tag)).toBeTruthy();
+    });
+  });
+
+  test('Should not duplicate existing tags', async () => {
+    const processor = buildProcessor({});
+
+    mockFetchImplementation();
+    const entity = baseEntity();
+    entity.metadata.tags = ['existing', 'tags', 'java'];
+
+    await processor.preProcessEntity(entity, null, null, null, cache);
+    expect(mockedFetch).toHaveBeenCalledTimes(1);
+    expect(entity.metadata.tags).toStrictEqual([
+      'existing',
+      'tags',
+      'java',
+      'c++',
+      'asp-dot-net',
+      'common-lisp',
+    ]);
+  });
+
+  test('Should not process Resource entities by default', async () => {
+    const processor = buildProcessor({});
+
+    mockFetchImplementation();
+    const entity = baseEntity();
+    entity.kind = 'Resource';
+
+    await processor.preProcessEntity(entity, null, null, null, cache);
+    expect(mockedFetch).toHaveBeenCalledTimes(0);
+    expect(entity.metadata.tags).toStrictEqual(undefined);
+  });
+
+  test('Can process Resource entities by overriding shouldProcessEntity', async () => {
+    const processor = buildProcessor({
+      shouldProcessEntity: (entity: Entity) => {
+        return entity.kind === 'Resource';
+      },
+    });
+
+    mockFetchImplementation();
+    const entity = baseEntity();
+    entity.kind = 'Resource';
+
+    await processor.preProcessEntity(entity, null, null, null, cache);
+    expect(mockedFetch).toHaveBeenCalledTimes(1);
+    expect(entity.metadata.tags).toStrictEqual([
+      'c++',
+      'asp-dot-net',
+      'java',
+      'common-lisp',
+    ]);
+  });
+
+  test('Can omit languages using languageMap', async () => {
+    const processor = buildProcessor({
+      languageMap: {
+        Java: '',
+        'ASP.net': '',
+      },
+    });
+
+    mockFetchImplementation();
+    const entity = baseEntity();
+    await processor.preProcessEntity(entity, null, null, null, cache);
+    expect(mockedFetch).toHaveBeenCalledTimes(1);
+    expect(entity.metadata.tags).toStrictEqual(['c++', 'common-lisp']);
+  });
+
+  test('Can rewrite langs using languageMap', async () => {
+    const processor = buildProcessor({
+      languageMap: {
+        Java: 'notjava',
+      },
+    });
+
+    mockFetchImplementation();
+    const entity = baseEntity();
+    await processor.preProcessEntity(entity, null, null, null, cache);
+    expect(mockedFetch).toHaveBeenCalledTimes(1);
+    expect(entity.metadata.tags).toStrictEqual([
+      'c++',
+      'asp-dot-net',
+      'notjava',
+      'common-lisp',
+    ]);
+  });
+
+  test('Can omit languages less than bytesThreshold', async () => {
+    const processor = buildProcessor({
+      bytesThreshold: 5000,
+    });
+
+    mockFetchImplementation();
+    const entity = baseEntity();
+    await processor.preProcessEntity(entity, null, null, null, cache);
+    expect(mockedFetch).toHaveBeenCalledTimes(1);
+    expect(entity.metadata.tags).toStrictEqual(['java', 'common-lisp']);
+  });
+
+  test('Can include languages that arent programming', async () => {
+    const processor = buildProcessor({
+      languageTypes: ['data'],
+    });
+
+    mockFetchImplementation();
+    const entity = baseEntity();
+    await processor.preProcessEntity(entity, null, null, null, cache);
+    expect(mockedFetch).toHaveBeenCalledTimes(1);
+    expect(entity.metadata.tags).toStrictEqual(['yaml', 'json']);
+  });
+
+  test('Refetches from API when cache disabled', async () => {
+    const processor = buildProcessor({
+      cacheTTL: { minutes: 0 },
+    });
+
+    mockFetchImplementation();
+    const entity = baseEntity();
+    await processor.preProcessEntity(entity, null, null, null, cache);
+    expect(mockedFetch).toHaveBeenCalledTimes(1);
+    expect(mockCacheGet).toHaveBeenCalledTimes(0);
+    expect(mockCacheSet).toHaveBeenCalledTimes(0);
+    mockedFetch.mockClear();
+    await processor.preProcessEntity(entity, null, null, null, cache);
+    expect(mockedFetch).toHaveBeenCalledTimes(1);
+    expect(mockCacheGet).toHaveBeenCalledTimes(0);
+    expect(mockCacheSet).toHaveBeenCalledTimes(0);
+  });
+
+  test('Caches across runs with cache enabled', async () => {
+    const processor = buildProcessor({
+      cacheTTL: { minutes: 5 },
+    });
+
+    mockFetchImplementation();
+    const entity = baseEntity();
+    await processor.preProcessEntity(entity, null, null, null, cache);
+    expect(mockedFetch).toHaveBeenCalledTimes(1);
+    expect(mockCacheGet).toHaveBeenCalledTimes(1);
+    expect(mockCacheSet).toHaveBeenCalledTimes(1);
+
+    mockedFetch.mockClear();
+    mockCacheGet.mockClear();
+    mockCacheSet.mockClear();
+    await processor.preProcessEntity(entity, null, null, null, cache);
+    expect(mockedFetch).toHaveBeenCalledTimes(0);
+    expect(mockCacheGet).toHaveBeenCalledTimes(1);
+    expect(mockCacheSet).toHaveBeenCalledTimes(0);
+  });
+});
+
+function mockFetchImplementation(): void {
+  mockedFetch.mockResolvedValue({
+    json: jest.fn().mockResolvedValue({
+      languageCount: 6,
+      totalBytes: 43823,
+      processedDate: '2023-06-20T21:37:48.337Z',
+      breakdown: [
+        {
+          name: 'YAML',
+          percentage: 2.23,
+          bytes: 979,
+          type: 'data',
+          color: '#cb171e',
+        },
+        {
+          name: 'JSON',
+          percentage: 1.31,
+          bytes: 574,
+          type: 'data',
+          color: '#292929',
+        },
+        {
+          name: 'C++',
+          percentage: 5.25,
+          bytes: 2300,
+          type: 'programming',
+          color: '#f34b7d',
+        },
+        {
+          name: 'ASP.net',
+          percentage: 6.97,
+          bytes: 3053,
+          type: 'programming',
+          color: '#178600',
+        },
+        {
+          name: 'Java',
+          percentage: 12.79,
+          bytes: 5603,
+          type: 'programming',
+          color: '#b07219',
+        },
+        {
+          name: 'Common Lisp',
+          percentage: 71.46,
+          bytes: 31314,
+          type: 'programming',
+          color: '#3fb68b',
+        },
+      ],
+    }),
+  } as unknown as Response);
+}
+
+function baseEntity(): Entity {
+  return {
+    apiVersion: 'backstage.io/v1alpha1',
+    kind: 'Component',
+    metadata: {
+      name: 'foo',
+    },
+  };
+}
+
+function buildProcessor(options: Partial<LinguistTagsProcessorOptions>) {
+  const config = new ConfigReader({
+    linguist: {
+      tagsProcessor: {
+        bytesThreshold: options.bytesThreshold,
+        languageTypes: options.languageTypes,
+        languageMap: options.languageMap,
+        cacheTTL: options.cacheTTL,
+      },
+    },
+  });
+  return LinguistTagsProcessor.fromConfig(config, {
+    logger: getVoidLogger(),
+    discovery,
+    shouldProcessEntity: options.shouldProcessEntity,
+  });
+}

plugins/linguist-backend/src/processor/LinguistTagsProcessor.ts

@@ -0,0 +1,269 @@
+/*
+ * Copyright 2023 The Backstage Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Entity, stringifyEntityRef } from '@backstage/catalog-model';
+import {
+  CatalogProcessor,
+  CatalogProcessorCache,
+} from '@backstage/plugin-catalog-node';
+import { DiscoveryService } from '@backstage/backend-plugin-api';
+import { Languages, LanguageType } from '@backstage/plugin-linguist-common';
+import fetch from 'node-fetch';
+import { Logger } from 'winston';
+import { HumanDuration, durationToMilliseconds } from '@backstage/types';
+import { Config } from '@backstage/config';
+
+/**
+ * A function which given an entity, determines if it should be processed for linguist tags.
+ * @public
+ */
+export type ShouldProcessEntity = (entity: Entity) => boolean;
+
+interface CachedData {
+  [key: string]: number | string[];
+  languages: string[];
+  cachedTime: number;
+}
+
+/**
+ * The constructor options for building the LinguistTagsProcessor
+ * @public
+ */
+export interface LinguistTagsProcessorOptions {
+  logger: Logger;
+  discovery: DiscoveryService;
+  /**
+   * Optional map that gives full control over which linguist languages should be included as tags and
+   * how they should be represented. The keys should be exact matches to languages in the linguist
+   * and the values should be how they render as backstage tags. Keep in mind that backstage has character
+   * requirements for tags. If you map a key to a falsey value, it will not be emitted as a tag.
+   */
+  languageMap?: Record<string, string | undefined>;
+  /**
+   * A function which determines which entities should be processed by the LinguistTagProcessor.
+   *
+   * The default is to process all entities of kind=Component
+   */
+  shouldProcessEntity?: ShouldProcessEntity;
+  /**
+   * Determines how long to cache language breakdowns for entities in the processor. Considering
+   * how often this processor runs, caching can help move some read traffic off of the linguist DB.
+   *
+   * If this caching is using up too much memory, you can disable it by setting cacheTTL to 0.
+   */
+  cacheTTL?: HumanDuration;
+  /**
+   * How many bytes must exist of a language in a repo before we consider it for adding a tag to
+   * the entity. This can be used if some repos have short utility scripts that may not be the primary
+   * language for the repo.
+   */
+  bytesThreshold?: number;
+  /**
+   * Which linguist file types to process tags for.
+   */
+  languageTypes?: LanguageType[];
+}
+
+/**
+ * This processor will fetch the language breakdown from the linguist API and
+ * add the languages to the entity as searchable tags.
+ *
+ * @public
+ * */
+export class LinguistTagsProcessor implements CatalogProcessor {
+  private logger: Logger;
+  private discovery: DiscoveryService;
+  private loggerMeta = { plugin: 'LinguistTagsProcessor' };
+  private languageMap: Record<string, string | undefined> = {};
+  private shouldProcessEntity: ShouldProcessEntity = (entity: Entity) => {
+    return entity.kind === 'Component';
+  };
+  private cacheTTLMilliseconds: number;
+  private bytesThreshold = 0;
+  private languageTypes: LanguageType[] = ['programming'];
+
+  getProcessorName(): string {
+    return 'LinguistTagsProcessor';
+  }
+
+  constructor(options: LinguistTagsProcessorOptions) {
+    this.logger = options.logger;
+    this.discovery = options.discovery;
+    if (options.shouldProcessEntity) {
+      this.shouldProcessEntity = options.shouldProcessEntity;
+    }
+    this.cacheTTLMilliseconds = durationToMilliseconds(
+      options.cacheTTL || { minutes: 30 },
+    );
+    if (options.bytesThreshold) {
+      this.bytesThreshold = options.bytesThreshold;
+    }
+    if (options.languageTypes) {
+      this.languageTypes = options.languageTypes;
+    }
+    if (options.languageMap) {
+      this.languageMap = options.languageMap;
+    }
+  }
+
+  static fromConfig(
+    config: Config,
+    options: LinguistTagsProcessorOptions,
+  ): LinguistTagsProcessor {
+    const c = config.getOptionalConfig('linguist.tagsProcessor');
+    if (c) {
+      options.bytesThreshold ??= c.getOptionalNumber('bytesThreshold');
+      options.languageTypes ??= c.getOptionalStringArray(
+        'languageTypes',
+      ) as LanguageType[];
+      options.languageMap ??= c.getOptional('languageMap');
+      options.cacheTTL ??= c.getOptional('cacheTTL');
+    }
+
+    return new LinguistTagsProcessor(options);
+  }
+
+  /**
+   * Given an entity ref, fetches the language breakdown from the Linguist backend HTTP API.
+   * @param entityRef - stringified entity ref
+   * @returns The language breakdown
+   */
+  private async getLanguagesFromLinguistAPI(
+    entityRef: string,
+  ): Promise<string[]> {
+    this.logger.debug(`Fetching languages from linguist API`, {
+      ...this.loggerMeta,
+      entityRef,
+    });
+
+    const baseUrl = await this.discovery.getBaseUrl('linguist');
+    const linguistApi = new URL(`${baseUrl}/entity-languages`);
+    linguistApi.searchParams.append('entityRef', entityRef);
+    const linguistData = await fetch(linguistApi).then(
+      res => res.json() as Promise<Languages>,
+    );
+    if (!linguistData || !linguistData.processedDate) {
+      return [];
+    }
+
+    return linguistData.breakdown
+      .filter(
+        b =>
+          this.languageTypes.includes(b.type) && b.bytes > this.bytesThreshold,
+      )
+      .map(b => b.name);
+  }
+
+  /**
+   * Cached wrapper around getLanguagesFromLinguistAPI
+   * @param cache - The CatalogProcessorCache
+   * @param entityRef - Stringified entity references
+   *
+   * @returns List of languages
+   */
+  private async getCachedLanguages(
+    cache: CatalogProcessorCache,
+    entityRef: string,
+  ): Promise<string[]> {
+    let cachedData = (await cache.get(entityRef)) as CachedData | undefined;
+    if (!cachedData || this.isExpired(cachedData)) {
+      const languages = await this.getLanguagesFromLinguistAPI(entityRef);
+      cachedData = { languages, cachedTime: Date.now() };
+      await cache.set(entityRef, cachedData);
+    }
+    this.logger.debug(`Fetched cached languages ${cachedData.languages}`, {
+      ...this.loggerMeta,
+      entityRef,
+    });
+    return cachedData.languages;
+  }
+
+  /**
+   * Determines if cached data is expired based on TTL
+   *
+   * @param cachedData - The cached data for this entity
+   * @returns True if data is expired
+   */
+  private isExpired(cachedData: CachedData): boolean {
+    const elapsed = Date.now() - (cachedData.cachedTime || 0);
+    return elapsed > this.cacheTTLMilliseconds;
+  }
+
+  /**
+   * This pre-processor will fetch linguist data for a Component and convert the language breakdown
+   * into entity tags which will be appended to the entity.
+   *
+   * @public
+   */
+  async preProcessEntity(
+    entity: Entity,
+    _: any,
+    __: any,
+    ___: any,
+    cache: CatalogProcessorCache,
+  ): Promise<Entity> {
+    if (!this.shouldProcessEntity(entity)) {
+      return entity;
+    }
+    const entityRef = stringifyEntityRef(entity);
+    this.logger.debug(`Processing ${entityRef}`, {
+      ...this.loggerMeta,
+      entityRef,
+    });
+
+    const languages =
+      this.cacheTTLMilliseconds > 0
+        ? await this.getCachedLanguages(cache, entityRef)
+        : await this.getLanguagesFromLinguistAPI(entityRef);
+
+    const tags = (entity.metadata.tags ||= []);
+    const originalTagCount = tags.length;
+
+    languages.forEach(lang => {
+      const cleanedUpLangTag =
+        lang in this.languageMap ? this.languageMap[lang] : sanitizeTag(lang);
+      if (cleanedUpLangTag && !tags.includes(cleanedUpLangTag)) {
+        tags.push(cleanedUpLangTag);
+      }
+    });
+
+    const addedCount = tags.length - originalTagCount;
+
+    this.logger.debug(`Added ${addedCount} language tags from linguist`, {
+      ...this.loggerMeta,
+      entityRef,
+    });
+
+    return entity;
+  }
+}
+
+/**
+ * Converts language tags from linguist to something acceptable by
+ * the tag requirements for backstage
+ *
+ * @param tag - A language tag from linguist
+ * @returns Cleaned up language tag
+ * @internal
+ */
+export function sanitizeTag(tag: string): string {
+  return tag
+    .toLowerCase()
+    .replace(/\.net/g, '-dot-net')
+    .replace(/[^a-z0-9:+#-]+/g, '-')
+    .replace(/-{2,}/g, '-')
+    .replace(/^-+|-+$/g, '');
+}

plugins/linguist-backend/src/processor/__snapshots__/LinguistTagsProcessor.test.ts.snap

@@ -0,0 +1,707 @@
+// Jest Snapshot v1, https://goo.gl/fbAQLP
+
+exports[`sanitizeTag Should clean up all linguist languages 1`] = `
+[
+  "1c-enterprise",
+  "2-dimensional-array",
+  "4d",
+  "abap",
+  "abap-cds",
+  "abnf",
+  "ags-script",
+  "aidl",
+  "al",
+  "ampl",
+  "antlr",
+  "api-blueprint",
+  "apl",
+  "asl",
+  "asn-1",
+  "asp-dot-net",
+  "ats",
+  "actionscript",
+  "ada",
+  "adblock-filter-list",
+  "adobe-font-metrics",
+  "agda",
+  "alloy",
+  "alpine-abuild",
+  "altium-designer",
+  "angelscript",
+  "ant-build-system",
+  "antlers",
+  "apacheconf",
+  "apex",
+  "apollo-guidance-computer",
+  "applescript",
+  "arc",
+  "asciidoc",
+  "aspectj",
+  "assembly",
+  "astro",
+  "asymptote",
+  "augeas",
+  "autohotkey",
+  "autoit",
+  "avro-idl",
+  "awk",
+  "basic",
+  "ballerina",
+  "batchfile",
+  "beef",
+  "befunge",
+  "berry",
+  "bibtex",
+  "bicep",
+  "bikeshed",
+  "bison",
+  "bitbake",
+  "blade",
+  "blitzbasic",
+  "blitzmax",
+  "bluespec",
+  "boo",
+  "boogie",
+  "brainfuck",
+  "brighterscript",
+  "brightscript",
+  "browserslist",
+  "c",
+  "c#",
+  "c++",
+  "c-objdump",
+  "c2hs-haskell",
+  "cap-cds",
+  "cil",
+  "clips",
+  "cmake",
+  "cobol",
+  "codeowners",
+  "collada",
+  "cson",
+  "css",
+  "csv",
+  "cue",
+  "cweb",
+  "cabal-config",
+  "cadence",
+  "cairo",
+  "cameligo",
+  "cap-n-proto",
+  "cartocss",
+  "ceylon",
+  "chapel",
+  "charity",
+  "checksums",
+  "chuck",
+  "circom",
+  "cirru",
+  "clarion",
+  "clarity",
+  "classic-asp",
+  "clean",
+  "click",
+  "clojure",
+  "closure-templates",
+  "cloud-firestore-security-rules",
+  "conll-u",
+  "codeql",
+  "coffeescript",
+  "coldfusion",
+  "coldfusion-cfc",
+  "common-lisp",
+  "common-workflow-language",
+  "component-pascal",
+  "cool",
+  "coq",
+  "cpp-objdump",
+  "creole",
+  "crystal",
+  "csound",
+  "csound-document",
+  "csound-score",
+  "cuda",
+  "cue-sheet",
+  "curry",
+  "cycript",
+  "cypher",
+  "cython",
+  "d",
+  "d-objdump",
+  "d2",
+  "digital-command-language",
+  "dm",
+  "dns-zone",
+  "dtrace",
+  "dafny",
+  "darcs-patch",
+  "dart",
+  "dataweave",
+  "debian-package-control-file",
+  "denizenscript",
+  "dhall",
+  "diff",
+  "directx-3d-file",
+  "dockerfile",
+  "dogescript",
+  "dotenv",
+  "dylan",
+  "e",
+  "e-mail",
+  "ebnf",
+  "ecl",
+  "eclipse",
+  "ejs",
+  "eq",
+  "eagle",
+  "earthly",
+  "easybuild",
+  "ecere-projects",
+  "ecmarkup",
+  "editorconfig",
+  "edje-data-collection",
+  "eiffel",
+  "elixir",
+  "elm",
+  "elvish",
+  "elvish-transcript",
+  "emacs-lisp",
+  "emberscript",
+  "erlang",
+  "euphoria",
+  "f#",
+  "f",
+  "figlet-font",
+  "flux",
+  "factor",
+  "fancy",
+  "fantom",
+  "faust",
+  "fennel",
+  "filebench-wml",
+  "filterscript",
+  "fluent",
+  "formatted",
+  "forth",
+  "fortran",
+  "fortran-free-form",
+  "freebasic",
+  "freemarker",
+  "frege",
+  "futhark",
+  "g-code",
+  "gaml",
+  "gams",
+  "gap",
+  "gcc-machine-description",
+  "gdb",
+  "gdscript",
+  "gedcom",
+  "glsl",
+  "gn",
+  "gsc",
+  "game-maker-language",
+  "gemfile-lock",
+  "gemini",
+  "genero",
+  "genero-forms",
+  "genie",
+  "genshi",
+  "gentoo-ebuild",
+  "gentoo-eclass",
+  "gerber-image",
+  "gettext-catalog",
+  "gherkin",
+  "git-attributes",
+  "git-config",
+  "git-revision-list",
+  "gleam",
+  "glyph",
+  "glyph-bitmap-distribution-format",
+  "gnuplot",
+  "go",
+  "go-checksums",
+  "go-module",
+  "go-workspace",
+  "godot-resource",
+  "golo",
+  "gosu",
+  "grace",
+  "gradle",
+  "grammatical-framework",
+  "graph-modeling-language",
+  "graphql",
+  "graphviz-dot",
+  "groovy",
+  "groovy-server-pages",
+  "haproxy",
+  "hcl",
+  "hlsl",
+  "hocon",
+  "html",
+  "html+ecr",
+  "html+eex",
+  "html+erb",
+  "html+php",
+  "html+razor",
+  "http",
+  "hxml",
+  "hack",
+  "haml",
+  "handlebars",
+  "harbour",
+  "haskell",
+  "haxe",
+  "hiveql",
+  "holyc",
+  "hosts-file",
+  "hy",
+  "hyphy",
+  "idl",
+  "igor-pro",
+  "ini",
+  "irc-log",
+  "idris",
+  "ignore-list",
+  "imagej-macro",
+  "imba",
+  "inform-7",
+  "ink",
+  "inno-setup",
+  "io",
+  "ioke",
+  "isabelle",
+  "isabelle-root",
+  "j",
+  "jar-manifest",
+  "jcl",
+  "jflex",
+  "json",
+  "json-with-comments",
+  "json5",
+  "jsonld",
+  "jsoniq",
+  "janet",
+  "jasmin",
+  "java",
+  "java-properties",
+  "java-server-pages",
+  "javascript",
+  "javascript+erb",
+  "jest-snapshot",
+  "jetbrains-mps",
+  "jinja",
+  "jison",
+  "jison-lex",
+  "jolie",
+  "jsonnet",
+  "julia",
+  "jupyter-notebook",
+  "just",
+  "krl",
+  "kaitai-struct",
+  "kakounescript",
+  "kerboscript",
+  "kicad-layout",
+  "kicad-legacy-layout",
+  "kicad-schematic",
+  "kickstart",
+  "kit",
+  "kotlin",
+  "kusto",
+  "lfe",
+  "llvm",
+  "lolcode",
+  "lsl",
+  "ltspice-symbol",
+  "labview",
+  "lark",
+  "lasso",
+  "latte",
+  "lean",
+  "less",
+  "lex",
+  "ligolang",
+  "lilypond",
+  "limbo",
+  "linker-script",
+  "linux-kernel-module",
+  "liquid",
+  "literate-agda",
+  "literate-coffeescript",
+  "literate-haskell",
+  "livescript",
+  "logos",
+  "logtalk",
+  "lookml",
+  "loomscript",
+  "lua",
+  "m",
+  "m4",
+  "m4sugar",
+  "matlab",
+  "maxscript",
+  "mdx",
+  "mlir",
+  "mql4",
+  "mql5",
+  "mtml",
+  "muf",
+  "macaulay2",
+  "makefile",
+  "mako",
+  "markdown",
+  "marko",
+  "mask",
+  "mathematica",
+  "maven-pom",
+  "max",
+  "mercury",
+  "mermaid",
+  "meson",
+  "metal",
+  "microsoft-developer-studio-project",
+  "microsoft-visual-studio-solution",
+  "minid",
+  "miniyaml",
+  "mint",
+  "mirah",
+  "modelica",
+  "modula-2",
+  "modula-3",
+  "module-management-system",
+  "monkey",
+  "monkey-c",
+  "moocode",
+  "moonscript",
+  "motoko",
+  "motorola-68k-assembly",
+  "move",
+  "muse",
+  "mustache",
+  "myghty",
+  "nasl",
+  "ncl",
+  "neon",
+  "nl",
+  "npm-config",
+  "nsis",
+  "nwscript",
+  "nasal",
+  "nearley",
+  "nemerle",
+  "netlinx",
+  "netlinx+erb",
+  "netlogo",
+  "newlisp",
+  "nextflow",
+  "nginx",
+  "nim",
+  "ninja",
+  "nit",
+  "nix",
+  "nu",
+  "numpy",
+  "nunjucks",
+  "nushell",
+  "oasv2-json",
+  "oasv2-yaml",
+  "oasv3-json",
+  "oasv3-yaml",
+  "ocaml",
+  "objdump",
+  "object-data-instance-notation",
+  "objectscript",
+  "objective-c",
+  "objective-c++",
+  "objective-j",
+  "odin",
+  "omgrofl",
+  "opa",
+  "opal",
+  "open-policy-agent",
+  "openapi-specification-v2",
+  "openapi-specification-v3",
+  "opencl",
+  "openedge-abl",
+  "openqasm",
+  "openrc-runscript",
+  "openscad",
+  "openstep-property-list",
+  "opentype-feature-file",
+  "option-list",
+  "org",
+  "ox",
+  "oxygene",
+  "oz",
+  "p4",
+  "pddl",
+  "peg-js",
+  "php",
+  "plsql",
+  "plpgsql",
+  "pov-ray-sdl",
+  "pact",
+  "pan",
+  "papyrus",
+  "parrot",
+  "parrot-assembly",
+  "parrot-internal-representation",
+  "pascal",
+  "pawn",
+  "pep8",
+  "perl",
+  "pic",
+  "pickle",
+  "picolisp",
+  "piglatin",
+  "pike",
+  "plantuml",
+  "pod",
+  "pod-6",
+  "pogoscript",
+  "polar",
+  "pony",
+  "portugol",
+  "postcss",
+  "postscript",
+  "powerbuilder",
+  "powershell",
+  "prisma",
+  "processing",
+  "procfile",
+  "proguard",
+  "prolog",
+  "promela",
+  "propeller-spin",
+  "protocol-buffer",
+  "protocol-buffer-text-format",
+  "public-key",
+  "pug",
+  "puppet",
+  "pure-data",
+  "purebasic",
+  "purescript",
+  "pyret",
+  "python",
+  "python-console",
+  "python-traceback",
+  "q#",
+  "qml",
+  "qmake",
+  "qt-script",
+  "quake",
+  "r",
+  "raml",
+  "rbs",
+  "rdoc",
+  "realbasic",
+  "rexx",
+  "rmarkdown",
+  "rpc",
+  "rpgle",
+  "rpm-spec",
+  "runoff",
+  "racket",
+  "ragel",
+  "raku",
+  "rascal",
+  "raw-token-data",
+  "rescript",
+  "readline-config",
+  "reason",
+  "reasonligo",
+  "rebol",
+  "record-jar",
+  "red",
+  "redcode",
+  "redirect-rules",
+  "regular-expression",
+  "ren-py",
+  "renderscript",
+  "rich-text-format",
+  "ring",
+  "riot",
+  "robotframework",
+  "roff",
+  "roff-manpage",
+  "rouge",
+  "routeros-script",
+  "ruby",
+  "rust",
+  "sas",
+  "scss",
+  "selinux-policy",
+  "smt",
+  "sparql",
+  "sqf",
+  "sql",
+  "sqlpl",
+  "srecode-template",
+  "ssh-config",
+  "star",
+  "stl",
+  "ston",
+  "svg",
+  "swig",
+  "sage",
+  "saltstack",
+  "sass",
+  "scala",
+  "scaml",
+  "scenic",
+  "scheme",
+  "scilab",
+  "self",
+  "shaderlab",
+  "shell",
+  "shellcheck-config",
+  "shellsession",
+  "shen",
+  "sieve",
+  "simple-file-verification",
+  "singularity",
+  "slash",
+  "slice",
+  "slim",
+  "smpl",
+  "smali",
+  "smalltalk",
+  "smarty",
+  "smithy",
+  "snakemake",
+  "solidity",
+  "soong",
+  "sourcepawn",
+  "spline-font-database",
+  "squirrel",
+  "stan",
+  "standard-ml",
+  "starlark",
+  "stata",
+  "stringtemplate",
+  "stylus",
+  "subrip-text",
+  "sugarss",
+  "supercollider",
+  "svelte",
+  "sway",
+  "swift",
+  "systemverilog",
+  "ti-program",
+  "tl-verilog",
+  "tla",
+  "toml",
+  "tsql",
+  "tsv",
+  "tsx",
+  "txl",
+  "talon",
+  "tcl",
+  "tcsh",
+  "tex",
+  "tea",
+  "terra",
+  "texinfo",
+  "text",
+  "textmate-properties",
+  "textile",
+  "thrift",
+  "turing",
+  "turtle",
+  "twig",
+  "type-language",
+  "typescript",
+  "unified-parallel-c",
+  "unity3d-asset",
+  "unix-assembly",
+  "uno",
+  "unrealscript",
+  "urweb",
+  "v",
+  "vba",
+  "vbscript",
+  "vcl",
+  "vhdl",
+  "vala",
+  "valve-data-format",
+  "velocity-template-language",
+  "verilog",
+  "vim-help-file",
+  "vim-script",
+  "vim-snippet",
+  "visual-basic-dot-net",
+  "visual-basic-6-0",
+  "volt",
+  "vue",
+  "vyper",
+  "wdl",
+  "wgsl",
+  "wavefront-material",
+  "wavefront-object",
+  "web-ontology-language",
+  "webassembly",
+  "webassembly-interface-type",
+  "webidl",
+  "webvtt",
+  "wget-config",
+  "whiley",
+  "wikitext",
+  "win32-message-file",
+  "windows-registry-entries",
+  "witcher-script",
+  "wollok",
+  "world-of-warcraft-addon-data",
+  "wren",
+  "x-bitmap",
+  "x-font-directory-index",
+  "x-pixmap",
+  "x10",
+  "xc",
+  "xcompose",
+  "xml",
+  "xml-property-list",
+  "xpages",
+  "xproc",
+  "xquery",
+  "xs",
+  "xslt",
+  "xojo",
+  "xonsh",
+  "xtend",
+  "yaml",
+  "yang",
+  "yara",
+  "yasnippet",
+  "yacc",
+  "yul",
+  "zap",
+  "zil",
+  "zeek",
+  "zenscript",
+  "zephir",
+  "zig",
+  "zimpl",
+  "curl-config",
+  "desktop",
+  "dircolors",
+  "ec",
+  "edn",
+  "fish",
+  "hoon",
+  "jq",
+  "kvlang",
+  "mirc-script",
+  "mcfunction",
+  "mupad",
+  "nanorc",
+  "nesc",
+  "ooc",
+  "q",
+  "restructuredtext",
+  "robots-txt",
+  "sed",
+  "wisp",
+  "xbase",
+]
+`;

plugins/linguist-backend/src/processor/index.ts

@@ -0,0 +1,20 @@
+/*
+ * Copyright 2023 The Backstage Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+export type {
+  LinguistTagsProcessorOptions,
+  ShouldProcessEntity,
+} from './LinguistTagsProcessor';
+export { LinguistTagsProcessor } from './LinguistTagsProcessor';

plugins/linguist-common/api-report.md

@@ -23,7 +23,7 @@ export type Language = {
   name: string;
   percentage: number;
   bytes: number;
-  type: string;
+  type: LanguageType;
   color?: `#${string}`;
 };
 
@@ -35,6 +35,9 @@ export type Languages = {
   breakdown: Language[];
 };
 
+// @public (undocumented)
+export type LanguageType = 'programming' | 'data' | 'markup' | 'prose';
+
 // @public (undocumented)
 export const LINGUIST_ANNOTATION = 'backstage.io/linguist';
 

plugins/linguist-common/src/types.ts

@@ -28,12 +28,15 @@ export type Languages = {
   breakdown: Language[];
 };
 
+/** @public */
+export type LanguageType = 'programming' | 'data' | 'markup' | 'prose';
+
 /** @public */
 export type Language = {
   name: string;
   percentage: number;
   bytes: number;
-  type: string;
+  type: LanguageType;
   color?: `#${string}`;
 };
 

yarn.lock

@@ -7734,6 +7734,7 @@ __metadata:
     "@backstage/config": "workspace:^"
     "@backstage/errors": "workspace:^"
     "@backstage/plugin-auth-node": "workspace:^"
+    "@backstage/plugin-catalog-node": "workspace:^"
     "@backstage/plugin-linguist-common": "workspace:^"
     "@backstage/types": "workspace:^"
     "@types/express": "*"
@@ -7741,6 +7742,7 @@ __metadata:
     express: ^4.18.1
     express-promise-router: ^4.1.0
     fs-extra: ^10.0.0
+    js-yaml: ^4.1.0
     knex: ^2.0.0
     linguist-js: ^2.5.3
     luxon: ^2.0.2