friso Download - friso Source code download

Friso is an open source, high-performance Chinese word segmenter developed in C language and implemented using the popular mmseg algorithm. Completely based on modular design and implementation, it can be easily implanted into other programs, such as MySQL, PHP, and provides plug-in implementations for php5, php7, ocaml, and lua. The source code can be compiled and used on various platforms without modification. After loading 200,000 entries, the memory usage is stable at 14.5M.

 Initialized in 0.088911sec
Mode: Complex
+-Version: 1.6.2 (UTF-8)
+---------------------------------------------------------------+
| Friso - a Chinese word segmentation writen by c.              |
| bug report email - [email protected].                   |
| or: visit https://github.com/lionsoul2014/friso.              |
|     java edition for https://github.com/lionsoul2014/jcseg    |
| type 'quit' to exit the program.                              |
+---------------------------------------------------------------+
friso>>

歧义和同义词:研究生命起源，混合词: 做B超检查身体，x射线本质是什么，今天去奇都ktv唱卡拉ok去，哆啦a梦是一个动漫中的主角，单位和全角: 2009年８月６日开始大学之旅，岳阳今天的气温为38.6℃, 也就是101.48℉, 英文数字: bug report [email protected] or visit http://code.google.com/p/jcseg, we all admire the hacker spirit!特殊数字: ① ⑩ ⑽ ㈩.

歧义 和 同义词 : 研究 琢磨 研讨 钻研 生命 起源 ， 混合词 : 做 b超 检查 身体 ， x射线 本质 是 什么 ， 今天 去 奇都ktv 唱 卡拉ok 去 ， 哆啦a梦 是 一个 动漫 中 的 主角 ， 单位 和 全角 : 2009年 8月 6日 开始 大学 之旅 ， 岳阳 今天 的 气温 为 38.6℃ , 也就是 101.48℉ , 英文 英语 数字 : bug report example gmail com [email protected] or visit http : / / code google com code.google.com / p / jcseg , we all admire appreciate like love enjoy the hacker spirit mind ! 特殊 数字 : .

make
sudo make install

# for testing
make testing

The configuration work to be done by Friso is very simple. Find the friso.ini configuration file and open it with a text editor.

 # friso configuration file.
#	do not change the name of the left key.
# @email	[email protected]
# @date		2012-12-20
#

# charset, only UTF8 and GBK support.
# set it with UTF8(0) or GBK(1)
friso.charset = 0

# lexicon directory absolute path.
#	the value must end with '/'
# this will tell friso how to find friso.lex.ini configuration file and all the lexicon files.
#
# if it is not start with '/' for linux, or matches no ':' for winnt in its value 
#	friso will search the friso.lex.ini relative to friso.ini
# absolute path search:
# linux:	friso.lex_dir = /c/products/friso/dict/UTF-8/
# Winnt:	friso.lex_dir = D:/products/friso/dict/UTF-8/
# relative path search (All system)
friso.lex_dir = ./vendors/dict/UTF-8/

# the maximum matching length.
friso.max_len = 5 

# 1 for recognition chinese name.
#	and 0 for closed it.
friso.r_name = 1

# the maximum length for the cjk words in a
#	chinese and english mixed word.
friso.mix_len = 2

# the maxinum length for the chinese last name adron.
friso.lna_len = 1

# append the synonyms words
friso.add_syn = 1

# clear the stopwords or not (1 to open it and 0 to close it)
# @date 2013-06-13
friso.clr_stw = 0

# keep the unrecongized words or not (1 to open it and 0 to close it)
# @date 2013-06-13
friso.keep_urec = 0

# use sphinx output style like 'admire|love|enjoy einsten'
# @date 2013-10-25
friso.spx_out = 0

# start the secondary segmentation for complex english token.
friso.en_sseg = 1

# min length of the secondary segmentation token. (better larger than 1)
friso.st_minl = 2

# default keep punctuations for english token.
friso.kpuncs = @%.#&+

# the threshold value for a char not a part of a chinese name.
friso.nthreshold = 2000000

# default mode for friso.
# 1 : simple mode - simply maxmum matching algorithm.
# 2 : complex mode - four rules of mmseg alogrithm.
# 3 : detect mode - only return the words that the do exists in the lexicon
friso.mode = 2

Friso currently provides word segmentation plug-ins for php5, php7, ocaml, and lua:

language	binding	author	state
php	php5-binding	dongyado	Completed
php	php7-binding	dongyado	Completed
ocaml	ocaml-binding	https://github.com/kandu	Completed
sphinx	sphinx-binding	lionsoul	Under development
lua	lua-binding	lionsoul	Under development

 /* 第一步：申明三个对象 */
friso_t friso ;          /* Friso 分词对象 */
friso_config_t config ;  /* Friso 配置对象 */
friso_task_t task ;      /* Friso 任务对象 */

/* 第二步：初始化相应的对象 */
friso  = friso_new ();
config = friso_new_config ();
task   = friso_new_task ();

/* 从friso.ini配置文件中初始化 friso */
if ( friso_init_from_ifile ( friso , config , "friso.ini文件地址" ) != 1 ) {
    /* friso 初始化失败 */
}

/*
 * 切分模式默认来自friso.ini中的设置
 * 可以通过friso_set_mode函数自定义切分模式(简易，复杂，检测模式)
 * 简易模式：__FRISO_SIMPLE_MODE__
 * 复杂模式：__FRISO_COMPLEX_MODE__
 * 检测模式：__FRISO_DETECT_MODE__
 * 例如，这里设置为使用复杂模式分词：
*/
friso_set_mode ( config , __FRISO_COMPLEX_MODE__ );


/* 第三步：设置分词内容 */
friso_set_text ( task , "分词的文本" );

/* 第四步：获取分词内容 */
while ( config -> next_token ( friso , config , task ) != NULL ) {
    /* 
      task存储了分词的结果，
      task->token->word: 词条内容
      task->token->offset: 词条在原始文本的offset
      task->token->length: 词条的长度(字节数)
      task->token->rlen: 词条的真正字节数(Friso转换后的长度-字节数)
    */
    printf ( "%s " , task -> token -> word );
}


/* 第五步：释放对象 */
friso_free_task ( task );
friso_free_config ( config );
friso_free ( friso );

Friso internally classifies the lexicon. Before managing the lexicon, you need to understand Friso's thesaurus categories. The classified integer values and their meanings are as follows:

 typedef enum {
 __LEX_CJK_WORDS__ = 0,     // 普通 CJK 词库
 __LEX_CJK_UNITS__ = 1,     // CJK 单位词库
 __LEX_ECM_WORDS__ = 2,     // 英中混合词(例如: b 超)
 __LEX_CEM_WORDS__ = 3,     // 中英混合词(例如: 卡拉 ok).
 __LEX_CN_LNAME__  = 4,     // 中文姓氏
 __LEX_CN_SNAME__  = 5,     // 中文单姓名词库
 __LEX_CN_DNAME1__ = 6,     // 中文双姓名首字词库
 __LEX_CN_DNAME2__ = 7,     // 中文双姓名尾字词库
 __LEX_CN_LNA__ = 8,        // 中文姓氏修饰词词库
 __LEX_STOPWORDS__ = 9,     // 停止词词库
 __LEX_ENPUN_WORDS__ = 10,  // 英文和标点混合词库(例如: c++)
 __LEX_OTHER_WORDS__ = 15,  // 无用
 __LEX_NCSYN_WORDS__ = 16   // 无用
} friso_lex_t;

The friso.lex.ini configuration file in the lexicon directory stores the lexicon category and the lexicon file name under the corresponding category. It is a one-to-many relationship. The default configuration is as follows:

 # main lexion
__LEX_CJK_WORDS__ :[
    lex-main.lex;
    lex-admin.lex;
    lex-chars.lex;
    lex-cn-mz.lex;
    lex-cn-place.lex;
    lex-company.lex;
    lex-festival.lex;
    lex-flname.lex;
    lex-food.lex;
    lex-lang.lex;
    lex-nation.lex;
    lex-net.lex;
    lex-org.lex;
    lex-touris.lex;
# add more here
]
# single chinese unit lexicon
__LEX_CJK_UNITS__ :[
    lex-units.lex;
]
# chinese and english mixed word lexicon like "b超".
__LEX_ECM_WORDS__:[
    lex-ecmixed.lex;
]
# english and chinese mixed word lexicon like "卡拉ok".
__LEX_CEM_WORDS__:[
    lex-cemixed.lex;
]
# chinese last name lexicon.
__LEX_CN_LNAME__:[
    lex-lname.lex;
]
# single name words lexicon.
__LEX_CN_SNAME__:[
    lex-sname.lex;
]
# first word of a double chinese name.
__LEX_CN_DNAME1__:[
    lex-dname-1.lex;
]
# second word of a double chinese name.
__LEX_CN_DNAME2__:[
    lex-dname-2.lex;
]
# chinese last name decorate word.
__LEX_CN_LNA__:[
    lex-ln-adorn.lex;
]
# stopwords lexicon
__LEX_STOPWORDS__:[
    lex-stopword.lex;
]
# english and punctuation mixed words lexicon.
__LEX_ENPUN_WORDS__:[
    lex-en-pun.lex;
]
# english words(for synonyms words)
__LEX_EN_WORDS__:[
    lex-en.lex;
]

 # main lexion
__LEX_CJK_WORDS__ :[
    lex-main.lex;
    lex-admin.lex;
    lex-chars.lex;
    lex-cn-mz.lex;
    lex-cn-place.lex;
    lex-company.lex;
    lex-festival.lex;
    lex-flname.lex;
    lex-food.lex;
    lex-lang.lex;
    lex-nation.lex;
    lex-net.lex;
    lex-org.lex;
    lex-touris.lex;
# 新增的植物名称词库
    lex-plants.lex;
# add more here
]

Find the corresponding thesaurus file, open it with a text editor, and add the terms to be added as one line in the following format (Note: It is recommended to confirm that the same terms do not exist before adding).