sitemapper

站点地图工具

Sitemapper 是一个用于生成 XML 站点地图的 Elixir 库。

它旨在生成大型站点地图，同时保持较低的内存配置文件。它可以将站点地图保存到 Amazon S3、GCP 存储、磁盘或您想要写入的任何其他适配器。

安装

 def deps do
  [
    { :sitemapper , "~> 0.8" }
  ]
end 

用法

  def generate_sitemap ( ) do
    config = [
      store: Sitemapper.S3Store ,
      store_config: [ bucket: "example-bucket" ] ,
      sitemap_url: "http://example-bucket.s3-aws-region.amazonaws.com"
    ]

    Stream . concat ( [ 1 .. 100_001 ] )
    |> Stream . map ( fn i ->
      % Sitemapper.URL {
        loc: "http://example.com/page- #{ i } " ,
        changefreq: :daily ,
        lastmod: Date . utc_today ( )
      }
    end )
    |> Sitemapper . generate ( config )
    |> Sitemapper . persist ( config )
    |> Stream . run ( )
  end

Sitemapper.generate接收 URL Stream 。这样可以轻松地将 Ecto Repo 的内容传输到站点地图中。

  def generate_sitemap ( ) do
    config = [
      store: Sitemapper.S3Store ,
      store_config: [ bucket: "example-bucket" ] ,
      sitemap_url: "http://example-bucket.s3-aws-region.amazonaws.com"
    ]

    Repo . transaction ( fn ->
      User
      |> Repo . stream ( )
      |> Stream . map ( fn % User { username: username , updated_at: updated_at } ->
        % Sitemapper.URL {
          loc: "http://example.com/users/ #{ username } " ,
          changefreq: :hourly ,
          lastmod: updated_at
        }
      end )
      |> Sitemapper . generate ( config )
      |> Sitemapper . persist ( config )
      |> Stream . run ( )
    end )
  end

要将站点地图保存到本地文件系统（而不是 Amazon S3），您的配置应如下所示：

 [
  store: Sitemapper.FileStore ,
  store_config: [
    path: sitemap_folder_path
  ] ,
  sitemap_url: "http://yourdomain.com"
]

请注意，您需要完成Stream.run/1或Enum.to_list/1才能执行流并返回结果。

托多

• 支持扩展的站点地图属性，如图像、视频等。

基准测试

sitemapper比fast_sitemap快约 50%。在下面的大型场景（1,000,000 个 URL）中， sitemapper使用约 50MB 峰值内存消耗，而fast_sitemap使用约 1000MB。

$ MIX_ENV=bench mix run bench/bench.exs
Operating System: macOS
CPU Information: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
Number of Available Cores: 8
Available memory: 16 GB
Elixir 1.9.1
Erlang 22.0.4

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 0 ns
parallel: 1
inputs: large, medium, small
Estimated total run time: 1.20 min

Benchmarking fast_sitemap - simple with input large...
Benchmarking fast_sitemap - simple with input medium...
Benchmarking fast_sitemap - simple with input small...
Benchmarking sitemapper - simple with input large...
Benchmarking sitemapper - simple with input medium...
Benchmarking sitemapper - simple with input small...

# #### With input large #####
Name                            ips        average  deviation         median         99th %
sitemapper - simple          0.0353        28.32 s     ±0.00%        28.32 s        28.32 s
fast_sitemap - simple        0.0223        44.76 s     ±0.00%        44.76 s        44.76 s

Comparison:
sitemapper - simple          0.0353
fast_sitemap - simple        0.0223 - 1.58x slower +16.45 s

# #### With input medium #####
Name                            ips        average  deviation         median         99th %
sitemapper - simple            0.35         2.85 s     ±0.57%         2.85 s         2.87 s
fast_sitemap - simple          0.23         4.28 s     ±0.46%         4.28 s         4.30 s

Comparison:
sitemapper - simple            0.35
fast_sitemap - simple          0.23 - 1.50x slower +1.43 s

# #### With input small #####
Name                            ips        average  deviation         median         99th %
sitemapper - simple           32.00       31.25 ms     ±8.01%       31.20 ms       37.22 ms
fast_sitemap - simple         21.55       46.41 ms     ±6.96%       46.26 ms       56.36 ms

Comparison:
sitemapper - simple           32.00
fast_sitemap - simple         21.55 - 1.49x slower +15.16 ms